==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== 41m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 41m Warning FailedMount pod/odh-observability-764b4d4657-jkbh5 MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 40m Warning FailedMount pod/model-registry-operator-controller-manager-65fc75f75f-z4gpv MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found 40m Warning FailedMount pod/rhods-dashboard-d77f668d7-s9gjl MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 40m Warning FailedMount pod/agent-ops-ui-694ff95d86-wxdgd MountVolume.SetUp failed for volume "proxy-tls" : secret "agent-ops-proxy-tls" not found 40m Warning FailedMount pod/automl-ui-74d9b97d49-25jfx MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 40m Warning FailedMount pod/model-registry-ui-667789dcc8-tcnjh MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 40m Warning FailedMount pod/maas-ui-bb55c885-2t4pc MountVolume.SetUp failed for volume "proxy-tls" : secret "maas-proxy-tls" not found 40m Warning FailedMount pod/autorag-ui-58f548cbd-8h2dd MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 40m Warning FailedMount pod/kserve-controller-manager-58c6879fc6-n2gv5 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-s9gjl Readiness probe failed: Get "https://10.133.0.48:8444/healthz": dial tcp 10.133.0.48:8444: connect: connection refused 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-s9gjl Readiness probe failed: Get "https://10.133.0.48:8943/healthcheck": dial tcp 10.133.0.48:8943: connect: connection refused 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-s9gjl Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-2tbwz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-2tbwz Readiness probe failed: Get "https://10.132.0.46:8943/healthcheck": dial tcp 10.132.0.46:8943: connect: connection refused 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-2tbwz Readiness probe failed: Get "https://10.132.0.46:8444/healthz": dial tcp 10.132.0.46:8444: connect: connection refused 41m Warning AppliedWithWarnings installplan/install-dbgwn 1 warning(s) generated during installation of operator "rhods-operator.3.6.0-ea.1" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 91s Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 4m30s Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 4m30s Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 5m4s Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 3m26s Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 5m4s Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 91s Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 3m26s Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 39m Warning FailedMount pod/model-catalog-7cb4dd5dc6-jxrxf MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 39m Warning Unhealthy pod/model-catalog-7cb4dd5dc6-jxrxf Readiness probe failed: Get "http://10.132.0.50:8080/readyz": dial tcp 10.132.0.50:8080: connect: connection refused 5m33s Warning FailedMount pod/e2e-test-registry-298031-7ff85855dd-dplbj MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-298031-kube-rbac-proxy-config" not found 4m2s Warning Unhealthy pod/default-db-registry-298031-848f9fcbb8-vt9jw Startup probe failed: Get "http://10.134.0.48:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 3m58s Warning Unhealthy pod/default-db-registry-298031-848f9fcbb8-vt9jw Readiness probe failed: HTTP probe failed with statuscode: 503 3m3s Warning Unhealthy pod/default-db-registry-298031-postgres-79f664c95c-skxpn Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2m27s Warning FailedMount pod/test-model-registry-662874-66cf8545bc-cdn6s MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "test-model-registry-662874-kube-rbac-proxy" not found 2m27s Warning FailedMount pod/test-model-registry-662874-66cf8545bc-cdn6s MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "test-model-registry-662874-kube-rbac-proxy" not found {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 MutatingWebhookConfiguration]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration]"} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"Error creating object, retrying...","reason":"Internal error occurred: failed calling webhook \"dscinitialization-v2-validator.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/validate-dscinitialization-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\""} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Error creating object, retrying...","reason":"Internal error occurred: failed calling webhook \"dscinitialization-v2-validator.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/validate-dscinitialization-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\""} {"level":"info","ts":"2026-09-11T12:37:20Z","msg":"Error creating object, retrying...","reason":"Internal error occurred: failed calling webhook \"dscinitialization-v2-validator.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/validate-dscinitialization-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\""} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:31Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","name":"monitoring","error":"no matches for kind \"Monitoring\" in version \"services.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:32Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:33Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 MutatingWebhookConfiguration]"} {"level":"error","ts":"2026-09-11T12:37:34Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:35Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:36Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:36Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:36Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-tls-rule: apply failed networking.istio.io/v1, Kind=DestinationRule: unable to patch networking.istio.io/v1, Kind=DestinationRule openshift-ingress/data-science-tls-rule: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:39Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:46Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","name":"dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","name":"modelregistry","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","name":"workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:50Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:50Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"; 1 more resources failed: /default-kserve (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"; 1 more resources failed: /default-kserve (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","name":"kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:54Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:54Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:54Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"; 1 more resources failed: /default-kserve (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:55Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:55Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-09-11T13:18:39Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.31 Kubernetes Version: v1.34.9 --- Namespaces --- namespace/opendatahub namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.3 Authorino Operator 1.4.3 authorino-operator.v1.4.2 Succeeded cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Succeeded custom-metrics-autoscaler.v2.19.0-3 Custom Metrics Autoscaler 2.19.0-3 custom-metrics-autoscaler.v2.19.0-2 Succeeded dns-operator.v1.4.2 DNS Operator 1.4.2 dns-operator.v1.4.1 Succeeded kueue-operator.v1.4.2 Red Hat build of Kueue 1.4.2 Succeeded limitador-operator.v1.4.2 Limitador Operator 1.4.2 limitador-operator.v1.4.1 Succeeded rhcl-operator.v1.4.3 Red Hat Connectivity Link 1.4.3 rhcl-operator.v1.4.2 Succeeded rhods-operator.3.6.0-ea.1 Red Hat OpenShift AI 3.6.0-ea.1 Succeeded serverless-operator.v1.37.1 Red Hat OpenShift Serverless 1.37.1 serverless-operator.v1.37.0 Succeeded --- CSVs not in Succeeded phase --- (all CSVs report phase Succeeded) --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aigateway: managementState=Removed aipipelines: managementState=Managed batchGateway: managementState=Removed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Removed llamastackoperator: managementState=n/a mcplifecycleoperator: managementState=Removed mlflowoperator: managementState=Removed modelregistry: managementState=Managed modelsAsAService: managementState=Removed ogx: managementState=Removed ray: managementState=Managed sparkoperator: managementState=Removed trainer: managementState=Removed trainingoperator: managementState=Removed trustyai: managementState=Removed workbenches: managementState=Managed workbenchesV2: managementState=Removed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= ModulesReady: status=True reason= message= AIGatewayReady: status=False reason=Removed message=Module ManagementState is set to Removed AIPipelinesReady: status=True reason= message= BatchGatewayReady: status=False reason=Removed message=Submodule ManagementState is set to Removed DashboardReady: status=True reason=AllDependentsHealthy message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=False reason=Removed message=Component ManagementState is set to Removed MCPLifecycleOperatorReady: status=False reason=Removed message=Module ManagementState is set to Removed MLflowOperatorReady: status=False reason=Removed message=Module ManagementState is set to Removed ModelRegistryReady: status=True reason=AllDependentsHealthy message= ModelsAsAServiceReady: status=False reason=Removed message=Submodule ManagementState is set to Removed OGXReady: status=False reason=Removed message=Module ManagementState is set to Removed RayReady: status=True reason= message= SparkOperatorReady: status=False reason=Removed message=Module ManagementState is set to Removed TrainerReady: status=False reason=Removed message=Module ManagementState is set to Removed TrustyAIReady: status=False reason=Removed message=Component ManagementState is set to Removed WorkbenchesReady: status=True reason=ReconcileSuccess message=Workbenches component is ready WorkbenchesV2Ready: status=False reason=Removed message=Submodule ManagementState is set to Removed --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 41m Ready 2026-09-11T12:37:28Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in Monitoring CR OpenTelemetryCollectorAvailable: status=False reason=MetricsAndTracesNotConfigured message=Metrics and traces are not configured in Monitoring CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in Monitoring CR NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR PersesAvailable: status=False reason=MetricsAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured ProvisioningSucceeded: status=True reason=Available message= Ready: status=True reason=Available message= --- Component CRs (Ready) --- NAME APPNAMESPACE AGE aihub.components.platform.opendatahub.io/default-aihub redhat-ods-applications 40m NAME PHASE READY REASON URL AGE dashboard.components.platform.opendatahub.io/default-dashboard Ready True AllDependentsHealthy https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/ 40m NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON PHASE workbenches.components.platform.opendatahub.io/default-workbenches True ReconcileSuccess Ready --- Deployments not fully ready (ODH/RHOAI namespaces) --- (all deployments report ready replicas == desired) --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- (all pods Running or Succeeded) --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/opendatahub --- (no events) --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 41m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 41m Warning FailedMount pod/odh-observability-764b4d4657-jkbh5 MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 40m Warning FailedMount pod/model-registry-operator-controller-manager-65fc75f75f-z4gpv MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found 40m Warning FailedMount pod/rhods-dashboard-d77f668d7-s9gjl MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 40m Warning FailedMount pod/agent-ops-ui-694ff95d86-wxdgd MountVolume.SetUp failed for volume "proxy-tls" : secret "agent-ops-proxy-tls" not found 40m Warning FailedMount pod/automl-ui-74d9b97d49-25jfx MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 40m Warning FailedMount pod/model-registry-ui-667789dcc8-tcnjh MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 40m Warning FailedMount pod/maas-ui-bb55c885-2t4pc MountVolume.SetUp failed for volume "proxy-tls" : secret "maas-proxy-tls" not found 40m Warning FailedMount pod/autorag-ui-58f548cbd-8h2dd MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 40m Warning FailedMount pod/kserve-controller-manager-58c6879fc6-n2gv5 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-s9gjl Readiness probe failed: Get "https://10.133.0.48:8444/healthz": dial tcp 10.133.0.48:8444: connect: connection refused 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-s9gjl Readiness probe failed: Get "https://10.133.0.48:8943/healthcheck": dial tcp 10.133.0.48:8943: connect: connection refused 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-s9gjl Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-2tbwz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-2tbwz Readiness probe failed: Get "https://10.132.0.46:8943/healthcheck": dial tcp 10.132.0.46:8943: connect: connection refused 38m Warning Unhealthy pod/rhods-dashboard-d77f668d7-2tbwz Readiness probe failed: Get "https://10.132.0.46:8444/healthz": dial tcp 10.132.0.46:8444: connect: connection refused --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- LAST SEEN TYPE REASON OBJECT MESSAGE 41m Warning AppliedWithWarnings installplan/install-dbgwn 1 warning(s) generated during installation of operator "rhods-operator.3.6.0-ea.1" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 91s Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 4m30s Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 4m30s Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 5m4s Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 3m26s Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 5m4s Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 91s Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 3m26s Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 39m Warning FailedMount pod/model-catalog-7cb4dd5dc6-jxrxf MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 39m Warning Unhealthy pod/model-catalog-7cb4dd5dc6-jxrxf Readiness probe failed: Get "http://10.132.0.50:8080/readyz": dial tcp 10.132.0.50:8080: connect: connection refused 5m33s Warning FailedMount pod/e2e-test-registry-298031-7ff85855dd-dplbj MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-298031-kube-rbac-proxy-config" not found 4m2s Warning Unhealthy pod/default-db-registry-298031-848f9fcbb8-vt9jw Startup probe failed: Get "http://10.134.0.48:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 3m58s Warning Unhealthy pod/default-db-registry-298031-848f9fcbb8-vt9jw Readiness probe failed: HTTP probe failed with statuscode: 503 3m3s Warning Unhealthy pod/default-db-registry-298031-postgres-79f664c95c-skxpn Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2m27s Warning FailedMount pod/test-model-registry-662874-66cf8545bc-cdn6s MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "test-model-registry-662874-kube-rbac-proxy" not found 2m27s Warning FailedMount pod/test-model-registry-662874-66cf8545bc-cdn6s MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "test-model-registry-662874-kube-rbac-proxy" not found --- namespace/rhods-notebooks --- LAST SEEN TYPE REASON OBJECT MESSAGE --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-09-11T12:03:40Z) --- redhat-ods-operator/rhods-operator-5c4c6d8cfd-bjlns --- {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"datasciencecluster.opendatahub.io/v1, Kind=DataScienceCluster"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"dscinitialization.opendatahub.io/v1, Kind=DSCInitialization"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v1"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v2"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-hardware-profile"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-prometheus-monitors"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-acceleratorprofile"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-hardwareprofile"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 MutatingWebhookConfiguration]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration]"} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"Error creating object, retrying...","reason":"Internal error occurred: failed calling webhook \"dscinitialization-v2-validator.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/validate-dscinitialization-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\""} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Error creating object, retrying...","reason":"Internal error occurred: failed calling webhook \"dscinitialization-v2-validator.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/validate-dscinitialization-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\""} {"level":"info","ts":"2026-09-11T12:37:20Z","msg":"Error creating object, retrying...","reason":"Internal error occurred: failed calling webhook \"dscinitialization-v2-validator.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/validate-dscinitialization-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\""} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:31Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","name":"monitoring","error":"no matches for kind \"Monitoring\" in version \"services.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:32Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:33Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 MutatingWebhookConfiguration]"} {"level":"error","ts":"2026-09-11T12:37:34Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:35Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:36Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:36Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:36Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-tls-rule: apply failed networking.istio.io/v1, Kind=DestinationRule: unable to patch networking.istio.io/v1, Kind=DestinationRule openshift-ingress/data-science-tls-rule: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:39Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:46Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","name":"dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","name":"modelregistry","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","name":"workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:50Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:50Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"; 1 more resources failed: /default-kserve (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"; 1 more resources failed: /default-kserve (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","name":"kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:54Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:54Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:54Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"; 1 more resources failed: /default-kserve (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:55Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:55Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} --- redhat-ods-operator/rhods-operator-5c4c6d8cfd-z6kcv --- {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"datasciencecluster.opendatahub.io/v1, Kind=DataScienceCluster"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"dscinitialization.opendatahub.io/v1, Kind=DSCInitialization"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-hardware-profile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-prometheus-monitors"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-acceleratorprofile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-hardwareprofile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} --- redhat-ods-operator/rhods-operator-5c4c6d8cfd-zrrwx --- {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"datasciencecluster.opendatahub.io/v1, Kind=DataScienceCluster"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"dscinitialization.opendatahub.io/v1, Kind=DSCInitialization"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-hardware-profile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-prometheus-monitors"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-acceleratorprofile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-hardwareprofile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-5c4c6d8cfd-bjlns.log ==================== {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"Cluster config","Operator Namespace":"redhat-ods-operator","Application Namespace":"redhat-ods-applications","Release":{"name":"OpenShift AI Self-Managed","version":"3.6.0-ea.1"},"Cluster":{"type":"OpenShift","version":"4.21.31"}} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"datasciencecluster.opendatahub.io/v1, Kind=DataScienceCluster"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"dscinitialization.opendatahub.io/v1, Kind=DSCInitialization"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v1"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v2"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-hardware-profile"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-prometheus-monitors"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-acceleratorprofile"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-hardwareprofile"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"creating reconciler","type":"service","name":"auth"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"creating reconciler","type":"service","name":"certconfigmapgenerator"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"creating reconciler","type":"service","name":"gateway"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"creating reconciler","type":"service","name":"setupcontroller"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"creating reconciler","type":"component","name":"datasciencepipelines"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"creating reconciler","type":"component","name":"ray"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"creating reconciler","type":"component","name":"kueue"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"creating reconciler","type":"component","name":"trustyai"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"starting manager"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/etc/metrics-certs/tls.crt","key":"/etc/metrics-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"tlssecurityprofilewatcher","source":"kind source: *v1.APIServer"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/etc/metrics-certs/tls.crt","key":"/etc/metrics-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"Attempting to acquire leader lease...","lock":"redhat-ods-operator/07ed84f7.opendatahub.io"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"Successfully acquired lease","lock":"redhat-ods-operator/07ed84f7.opendatahub.io"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[networking.k8s.io/v1 NetworkPolicy]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 Role]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[dscinitialization.opendatahub.io/v2 DSCInitialization]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 RoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[v1 Namespace]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRole]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[v1 Secret]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[apps/v1 Deployment]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[v1 ServiceAccount]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[v1 Service]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[route.openshift.io/v1 Route]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[v1 PersistentVolumeClaim]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 ValidatingAdmissionPolicy]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 ValidatingAdmissionPolicyBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[infrastructure.opendatahub.io/v1 HardwareProfile]"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"run upgrade task"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[services.platform.opendatahub.io/v1alpha1 Auth]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[services.platform.opendatahub.io/v1alpha1 GatewayConfig]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[config.opendatahub.io/v1alpha1 Platform]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 TrustyAI]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRole]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 Role]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 RoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[v1 Service]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[apps/v1 Deployment]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","source":"kind source: *unstructured.Unstructured[v1 Namespace]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[datasciencecluster.opendatahub.io/v2 DataScienceCluster]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[dscinitialization.opendatahub.io/v2 DSCInitialization]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[services.platform.opendatahub.io/v1alpha1 GatewayConfig]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","source":"kind source: *unstructured.Unstructured[services.platform.opendatahub.io/v1alpha1 Auth]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRole]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 Role]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[gateway.networking.k8s.io/v1 HTTPRoute]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 RoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","source":"kind source: *unstructured.Unstructured[v1 Namespace]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","source":"kind source: *unstructured.Unstructured[v1 ServiceAccount]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[v1 Secret]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"cert-configmap-generator-controller","source":"kind source: *unstructured.Unstructured[dscinitialization.opendatahub.io/v2 DSCInitialization]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[services.platform.opendatahub.io/v1alpha1 GatewayConfig]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[oauth.openshift.io/v1 OAuthClient]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[gateway.networking.k8s.io/v1 GatewayClass]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[gateway.networking.k8s.io/v1 Gateway]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[v1 Secret]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"cert-configmap-generator-controller","source":"kind source: *unstructured.Unstructured[v1 Namespace]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"cert-configmap-generator-controller","source":"kind source: *v1.PartialObjectMetadata[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[v1 Service]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[apps/v1 Deployment]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[autoscaling/v2 HorizontalPodAutoscaler]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[gateway.networking.k8s.io/v1 HTTPRoute]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[route.openshift.io/v1 Route]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 DataSciencePipelines]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[v1 Secret]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRole]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 Role]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 RoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[v1 ServiceAccount]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[v1 Service]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[monitoring.coreos.com/v1 ServiceMonitor]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[apps/v1 Deployment]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","source":"kind source: *unstructured.Unstructured[security.openshift.io/v1 SecurityContextConstraints]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[services.platform.opendatahub.io/v1alpha1 Auth]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 Role]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Kueue]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[monitoring.coreos.com/v1 PrometheusRule]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 MutatingWebhookConfiguration]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[apps/v1 Deployment]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[v1 Secret]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRole]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Ray]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[operators.coreos.com/v2 OperatorCondition]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[v1 ServiceAccount]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[v1 Service]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[networking.k8s.io/v1 NetworkPolicy]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[monitoring.coreos.com/v1 PodMonitor]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 RoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:13Z","msg":"Starting EventSource","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:13Z","logger":"setup","msg":"create default DSCI CR"} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"create default DSCI CR."} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"Error creating object, retrying...","reason":"Internal error occurred: failed calling webhook \"dscinitialization-v2-validator.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/validate-dscinitialization-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\""} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"tlssecurityprofilewatcher"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"tlssecurityprofilewatcher","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"cert-configmap-generator-controller"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"cert-configmap-generator-controller","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"reconcile","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"d26e2dab-ac34-41ab-a031-de3bc9bd76c8"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting Controller","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Starting workers","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","worker count":1} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"8c74e58d-5b19-4fe1-8cb0-d01fd0b0e6e6"} {"level":"info","ts":"2026-09-11T12:37:16Z","msg":"Error creating object, retrying...","reason":"Internal error occurred: failed calling webhook \"dscinitialization-v2-validator.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/validate-dscinitialization-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\""} {"level":"info","ts":"2026-09-11T12:37:20Z","msg":"Error creating object, retrying...","reason":"Internal error occurred: failed calling webhook \"dscinitialization-v2-validator.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/validate-dscinitialization-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\""} {"level":"info","ts":"2026-09-11T12:37:28Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"35abda0c-8437-47a2-8015-3aaa8a9bfcef","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-09-11T12:37:28Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"knative-eventing","reconcileID":"853ef71e-c0cb-4a8d-a5ff-cad4b2c8024a"} {"level":"info","ts":"2026-09-11T12:37:28Z","logger":"DSCInitialization","msg":"Adding finalizer for DSCInitialization","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"35abda0c-8437-47a2-8015-3aaa8a9bfcef","name":"default-dsci","finalizer":"dscinitialization.opendatahub.io/finalizer"} {"level":"info","ts":"2026-09-11T12:37:28Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"knative-serving","reconcileID":"11a757e5-6f27-4554-81fc-e1b99ae650db"} {"level":"info","ts":"2026-09-11T12:37:28Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"odh-model-registries","reconcileID":"d3fd78bd-b34a-4867-8920-2e4df4169266"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cert-manager-operator","reconcileID":"c97c95bb-9373-441b-947c-1273e60d04a5"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"kuadrant-system","reconcileID":"748b10bf-5064-4a52-830f-94223800feef"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"35abda0c-8437-47a2-8015-3aaa8a9bfcef","name":"redhat-ods-applications"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"opendatahub","reconcileID":"7a6f6696-c56c-463a-8a98-f045e375df01"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"redhat-ods-applications","reconcileID":"8b9e0143-4957-423f-8946-6ae61ed466a8"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cert-manager","reconcileID":"7e79ac45-360f-4abb-9fdf-a8798390f810"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"redhat-ods-operator","reconcileID":"dc87d651-53a4-48de-a740-c6b9a63aec38"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"redhat-ods-monitoring","reconcileID":"74ef4e87-d8b2-4045-803b-e5a1a6c29d9a"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"rhoai-model-registries","reconcileID":"09790685-108a-4085-8b70-d1dc05121b94"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"kyverno","reconcileID":"0e8c5a95-ee0c-4175-9fb2-516fd620b8fb"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"knative-serving-ingress","reconcileID":"3d5a2171-9813-45a8-8924-9aa800c61a0c"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"knative-eventing","reconcileID":"f7ca9f63-e802-4569-9a2a-8c2b011657c8"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"knative-serving","reconcileID":"619e4f9c-f505-425a-96c3-f7fffb97d2bb"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"odh-model-registries","reconcileID":"7a729d4c-8c33-4a22-a490-35e33b526ef3"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"reconcile","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"apply","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.initialize"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.createDefaultGroup"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cert-manager-operator","reconcileID":"901feeac-235c-4ade-8dcb-a8457b330f85"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"kuadrant-system","reconcileID":"b10b9044-7fc7-4366-9f81-010b68195896"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"adding finalizer","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"opendatahub","reconcileID":"51b63e82-71af-472e-ab26-480990a7167e"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"redhat-ods-applications","reconcileID":"2ca98c69-7842-4ddc-8ced-51fd0222ea94"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.managePermissions"} {"level":"info","ts":"2026-09-11T12:37:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.managePermissions","msg":"namespace not found, skipping RoleBinding creation","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f","namespace":"models-as-a-service","roleBinding":"data-science-admingroup-maas-rolebinding"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:29Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2c5f1422-f5d1-442d-9f05-59f549fa4e2b","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-09-11T12:37:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cert-manager","reconcileID":"32d3ab1f-f65a-40a5-9628-3d3d1ab77c20"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2c5f1422-f5d1-442d-9f05-59f549fa4e2b","name":"redhat-ods-applications"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"redhat-ods-operator","reconcileID":"b13a4c35-f6f4-4934-babd-aab11464548e"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"redhat-ods-monitoring","reconcileID":"809cd833-96c7-4129-9933-7fbac5eed149"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"rhoai-model-registries","reconcileID":"9bc354c7-4b46-4cd2-b462-4e9d77bc76cd"} {"level":"info","ts":"2026-09-11T12:37:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"ogx"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"kyverno","reconcileID":"de1fc743-44d9-4b37-a75a-56b3d7c70842"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"knative-serving-ingress","reconcileID":"bc063e6d-58f3-4ab1-a02c-0e7ac65c3942"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"kyverno","reconcileID":"ac99fedf-9d23-4bc2-b68b-8e81466c03ec"} {"level":"info","ts":"2026-09-11T12:37:29Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"15032849-d0f3-4727-9e2b-c8a58a8c1dea","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"knative-serving-ingress","reconcileID":"e4b91871-7502-436e-873b-fcaf2333997f"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"11599bab-4bd7-4941-ac26-c4da01d0d332","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5b441785-3f51-4d05-9cd5-4e71a95b99b2","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f57be5b8-ec5c-42b3-afa1-894afa209e4a","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:30Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95af7e82-3779-4a10-8c6f-855aa3292452","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"feastoperator"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:31Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d17322e-f791-4d46-9027-5afa0d7286e8","error":"provisioning failed: failed to lookup object openshift-ingress/data-science-tls-rule: no matches for kind \"DestinationRule\" in version \"networking.istio.io/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca"} {"level":"info","ts":"2026-09-11T12:37:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"kserve"} {"level":"info","ts":"2026-09-11T12:37:31Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm","msg":"Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f"} {"level":"info","ts":"2026-09-11T12:37:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"workbenches"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"c0212e83-d08f-4260-a8a1-421ee2e07c5f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"reconcile","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"apply","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.initialize"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.createDefaultGroup"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.managePermissions"} {"level":"info","ts":"2026-09-11T12:37:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.managePermissions","msg":"namespace not found, skipping RoleBinding creation","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e","namespace":"models-as-a-service","roleBinding":"data-science-admingroup-maas-rolebinding"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:31Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"caacf8d8-60b7-49ca-8b3f-edd306db963e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"trainer"} {"level":"info","ts":"2026-09-11T12:37:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"modelregistry"} {"level":"info","ts":"2026-09-11T12:37:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:37:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"dashboard"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:37:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","module":"monitoring","runlevel":"20"} {"level":"error","ts":"2026-09-11T12:37:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","name":"monitoring","error":"no matches for kind \"Monitoring\" in version \"services.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"runlevel gating: waiting for prior entries","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","runlevel":"31","blocked_on":["monitoring"],"elapsed":0,"timeout":600,"requeue_after":599} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:32Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"acf02161-8cea-461a-87ea-3045ccb147d7","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"acf02161-8cea-461a-87ea-3045ccb147d7","name":"redhat-ods-applications"} {"level":"error","ts":"2026-09-11T12:37:32Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c82593b7-3842-411b-abb0-d9d2d7f3eaca","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:32Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"4d53152d-ebd6-4bfa-b7d2-bdf5a7bf75a9","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Starting EventSource","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","source":"kind source: *unstructured.Unstructured[services.platform.opendatahub.io/v1alpha1 Monitoring]"} {"level":"info","ts":"2026-09-11T12:37:32Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"4d53152d-ebd6-4bfa-b7d2-bdf5a7bf75a9","name":"redhat-ods-applications"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:37:33Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"7cf3a279-10af-4f7d-83a1-736f8a480a3d","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"7cf3a279-10af-4f7d-83a1-736f8a480a3d","name":"redhat-ods-applications"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:33Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:33Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"88ccc3ce-a8f3-439a-b55e-28f1a74616ac","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 DataSciencePipelines]"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Ray]"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Kueue]"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 TrustyAI]"} {"level":"info","ts":"2026-09-11T12:37:33Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[services.platform.opendatahub.io/v1alpha1 Monitoring]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"decd2ad6-4efa-415b-9482-d9357adaed0a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[v1 ServiceAccount]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRole]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 ClusterRoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[v1 ConfigMap]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[v1 Service]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[cert-manager.io/v1 Issuer]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[cert-manager.io/v1 Certificate]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[apps/v1 Deployment]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 MutatingWebhookConfiguration]"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4"} {"level":"info","ts":"2026-09-11T12:37:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"sparkoperator"} {"level":"error","ts":"2026-09-11T12:37:34Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"83d571f4-8633-494e-8ba0-109bdfdabf25","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"ogx"} {"level":"info","ts":"2026-09-11T12:37:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:37:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"feastoperator"} {"level":"info","ts":"2026-09-11T12:37:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:35Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c1459799-b57b-4e50-9922-d41c4d2707c6","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"kserve"} {"level":"info","ts":"2026-09-11T12:37:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"workbenches"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"trainer"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"modelregistry"} {"level":"error","ts":"2026-09-11T12:37:36Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"335f76d4-e33e-48d0-a0aa-0105292eeff8","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:36Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3a5ddeb-2f62-485a-b54a-b4e0549f4ea5","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-authn-filter: apply failed networking.istio.io/v1alpha3, Kind=EnvoyFilter: unable to patch networking.istio.io/v1alpha3, Kind=EnvoyFilter openshift-ingress/data-science-authn-filter: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure.detectAndSetIngressMode","msg":"Detected ingress mode from existing Gateway Service","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","mode":"OcpRoute"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"dashboard"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:37:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:36Z","msg":"Reconciler error","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"418efd3a-eeaf-4910-b21a-f12fbcfb4b0b","error":"provisioning failed: failure deploying resource openshift-ingress/data-science-tls-rule: apply failed networking.istio.io/v1, Kind=DestinationRule: unable to patch networking.istio.io/v1, Kind=DestinationRule openshift-ingress/data-science-tls-rule: Internal error occurred: failed calling webhook \"rev.validation.istio.io\": failed to call webhook: Post \"https://istiod-openshift-gateway.openshift-ingress.svc:443/validate?timeout=10s\": no endpoints available for service \"istiod-openshift-gateway\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"EnvoyFilter exposes internal implementation details that may change at any time. Prefer other APIs if possible, and exercise extreme caution, especially around upgrades.","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0710159-1ab2-4bbc-b11a-268230fb45d7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[networking.istio.io/v1alpha3 EnvoyFilter]"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[networking.istio.io/v1 DestinationRule]"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[config.openshift.io/v1 APIServer]"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4098b2a6-f923-49fa-9cf1-93324bd1c0b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"dc1e1648-b8af-4a92-a651-1d5c6d202001","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:37:38Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"0b5be1f3-e017-4ed2-a41f-3acd754dee84","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"0b5be1f3-e017-4ed2-a41f-3acd754dee84","name":"redhat-ods-applications"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"adding finalizer","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70"} {"level":"info","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"ogx"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:38Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"10c84545-def2-44ed-99e8-40b0687bd71c","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"10c84545-def2-44ed-99e8-40b0687bd71c","name":"redhat-ods-applications"} {"level":"info","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","modules":"dashboard, feastoperator, kserve, modelregistry, workbenches"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"reconcile","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"apply","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/precondition.runlevelGateAction"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.initialize"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.argoWorkflowsControllersOptions"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} # Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"reconcile","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025"} {"level":"info","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"apply","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/precondition.runlevelGateAction"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/sanitycheck.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ray.initialize"} {"level":"error","ts":"2026-09-11T12:37:38Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:38Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:37:39Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:39Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c62f9ae0-9b5b-4fc2-b117-1f9b7b22bc70","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:39Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630"} {"level":"info","ts":"2026-09-11T12:37:39Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630"} {"level":"info","ts":"2026-09-11T12:37:39Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:39Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","modules":"dashboard, feastoperator, kserve, modelregistry, workbenches"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:40Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"feastoperator"} {"level":"error","ts":"2026-09-11T12:37:40Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7f11c2f7-7a00-45a6-b3d4-4b2d01f5d630","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a"} {"level":"info","ts":"2026-09-11T12:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","modules":"dashboard, feastoperator, kserve, modelregistry, workbenches"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"aigateway"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:41Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"16f9a056-c88a-433f-a3ff-973d6ffcfd8a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-09-11T12:37:41Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3abe9da3-87b9-4947-a235-7969b6ba019e","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"kserve"} {"level":"info","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","modules":"dashboard, feastoperator, kserve, modelregistry, workbenches"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"workbenches"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e4afa6c8-9117-44f4-9412-39db95477edc","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-09-11T12:37:43Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4db39ce3-b3b3-495e-961f-a5db37ae24e9","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a713b0c5-9ce1-4448-8dd4-199f6614b121","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"trainer"} {"level":"info","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","modules":"dashboard, feastoperator, kserve, modelregistry, workbenches"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:44Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"774fbf05-e715-45e3-85c1-719b9097a7c1","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"modelregistry"} {"level":"info","ts":"2026-09-11T12:37:44Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","modules":"dashboard, feastoperator, kserve, modelregistry, workbenches"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:45Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"0da38c8a-39f8-4fb9-9c70-c58fa24307fa","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"0da38c8a-39f8-4fb9-9c70-c58fa24307fa","name":"redhat-ods-applications"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CRD not registered, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"mcplifecycleoperator"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1a2bd512-8b83-4384-be97-54b3fbb28baa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-09-11T12:37:45Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ee879374-9918-4287-8a11-2925a53438be","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"dashboard"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:37:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:46Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm","msg":"Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d"} {"level":"info","ts":"2026-09-11T12:37:46Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","modules":"dashboard, feastoperator, kserve, modelregistry, workbenches"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"584b1168-1138-40ff-864e-95016b133fc9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:46Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:46Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"7df072b8-f635-46ff-886b-8607e434ae7d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"reconcile","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"apply","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/precondition.runlevelGateAction"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.initialize"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.argoWorkflowsControllersOptions"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:47Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fd950e7c-9a8b-4ec3-b5cd-5bef19822202","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"ogx"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:47Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","modules":"dashboard, feastoperator, kserve, modelregistry, workbenches"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-aihub","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/status/deployments.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","namespace":"","name":"default-workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"8c4bfa72-22d0-46dc-a72a-1aa2d47c9ef5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-09-11T12:37:48Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a85892c2-74cd-4fa3-9a73-4e3bb5a649e7","error":"provisioning failed: failed to lookup object /default-aihub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"; 4 more resources failed: /default-dashboard, /default-feastoperator, /default-kserve, /default-workbenches (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"trainer"} {"level":"info","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","name":"dashboard","error":"no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","name":"modelregistry","error":"no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","name":"workbenches","error":"no matches for kind \"Workbenches\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"runlevel gating: waiting for prior entries","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","runlevel":"31","blocked_on":["dashboard","modelregistry","workbenches"],"elapsed":0,"timeout":600,"requeue_after":599} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:37:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","modules":"dashboard, feastoperator, kserve, modelregistry, workbenches"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:49Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CR not found, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"37c694ff-3349-4e57-a6e8-766f8eab69ca","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Starting EventSource","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Dashboard]"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:49Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Dashboard CR not found, cleaning up dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bcfed742-8cce-4215-b330-faa4f2ebb447","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm","msg":"Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"04354752-4d66-4e85-bf45-5515af3eb025","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"reconcile","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"apply","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/precondition.runlevelGateAction"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/sanitycheck.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ray.initialize"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:49Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"error","ts":"2026-09-11T12:37:50Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:50Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:51Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"02b3fe8d-3be7-4d21-8757-72e4dda44a2c","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"; 1 more resources failed: /default-kserve (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:51Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","modules":"feastoperator, kserve"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:51Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:51Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"57199f21-e20a-4784-8a67-5ac39c3e0150","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"; 1 more resources failed: /default-kserve (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Dashboard]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 AIHub]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Workbenches]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4e1fb793-a6d7-4c10-88e4-ff641fac3541","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 Role]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[rbac.authorization.k8s.io/v1 RoleBinding]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[monitoring.coreos.com/v1 ServiceMonitor]"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:37:51Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:37:51Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:37:52Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"ogx"} {"level":"info","ts":"2026-09-11T12:37:52Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm","msg":"Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7"} {"level":"info","ts":"2026-09-11T12:37:52Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:37:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"537745c6-7ae7-40b6-b9aa-e9d229f4eff7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:52Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd"} {"level":"info","ts":"2026-09-11T12:37:52Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd"} {"level":"info","ts":"2026-09-11T12:37:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"73472593-267b-4045-8939-4d049aa6b8fd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2a179866-4189-4a28-9e89-f74e051664e4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"trainer"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","module":"kserve","runlevel":"31"} {"level":"error","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"readiness check failed, treating as not ready","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","name":"kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.WalkBatches\n\t/workspace/pkg/controller/provision/gating.go:80\ngithub.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules\n\t/workspace/internal/controller/modules/modules_controller_actions.go:360\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"runlevel gating: waiting for prior entries","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","runlevel":"32","blocked_on":["kserve"],"elapsed":0,"timeout":600,"requeue_after":599} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:54Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","modules":"feastoperator, kserve"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:54Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090"} {"level":"error","ts":"2026-09-11T12:37:54Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","namespace":"","name":"default-kserve","error":"no matches for kind \"Kserve\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:54Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"error","ts":"2026-09-11T12:37:54Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"94eb6ee4-e5f4-4822-8538-a21beace1649","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"; 1 more resources failed: /default-kserve (see logs for details)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74c9d375-cf47-4522-921c-c07ad14bf090","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:54Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:37:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a3cc19c-46e5-47f0-9e80-e54acef5157a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus","msg":"module CRDs not yet available, requesting requeue","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","modules":"feastoperator"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"error","ts":"2026-09-11T12:37:55Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"resource lookup failed, continuing","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","namespace":"","name":"default-feastoperator","error":"no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/actions/deploy/action_deploy.go:260\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).apply\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:504\ngithub.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*Reconciler).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/github.com/opendatahub-io/odh-platform-utilities/framework@v0.0.0-20260810103007-dd94df3e806f/controller/reconciler/reconciler.go:363\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"error","ts":"2026-09-11T12:37:55Z","msg":"Reconciler error","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"918e3e8e-a931-41ac-a913-c7c5e915537a","error":"provisioning failed: failed to lookup object /default-feastoperator: no matches for kind \"FeastOperator\" in version \"components.platform.opendatahub.io/v1alpha1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Kserve]"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"rhods-notebooks","reconcileID":"2c18ac70-7dde-48b6-9b0d-23702533011a"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dbd8d12-ff8e-43ae-a5a3-aeb6d42487fd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 ValidatingAdmissionPolicy]"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[admissionregistration.k8s.io/v1 ValidatingAdmissionPolicyBinding]"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"rhods-notebooks","reconcileID":"7695bc17-4058-44ba-93a0-87387c756d95"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"78438eac-c961-4329-9e87-aef5dee20b5f"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:37:56Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:37:56Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:37:56Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"ogx"} {"level":"info","ts":"2026-09-11T12:37:57Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:37:57Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:37:57Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"trainer"} {"level":"info","ts":"2026-09-11T12:37:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:37:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:37:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:37:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"reconcile","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"apply","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/precondition.runlevelGateAction"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.initialize"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.argoWorkflowsControllersOptions"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"ff42295d-dcee-44db-b45d-7c660bc68c80","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Dashboard]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 FeastOperator]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Kserve]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 AIHub]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Workbenches]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"77ac97d2-d7d8-4bb4-9b7e-c3c260e69ad8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 AIHub]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Dashboard]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 DataSciencePipelines]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 FeastOperator]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Kserve]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Ray]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Starting EventSource","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 Workbenches]"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:37:59Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:00Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b160daca-8b35-4eb0-806b-461b3d540c48","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[components.platform.opendatahub.io/v1alpha1 FeastOperator]"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f001e0d8-5ed6-46a1-9077-07673eb5b088","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"fa9c8c60-8f4d-4057-abab-4ca62b004fe4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Starting EventSource","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","source":"kind source: *unstructured.Unstructured[apiextensions.k8s.io/v1 CustomResourceDefinition]"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"341aff91-c014-4d36-9946-bc62004da728"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:00Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:00Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:00Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:00Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"reconcile","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"apply","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/precondition.runlevelGateAction"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/sanitycheck.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ray.initialize"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"f4a0287e-81f9-47e8-9d20-99f1ddb11cef","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:01Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"511ef9c3-6251-416b-9b13-135b4640a087","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe3e97c3-36fe-4099-81fd-83dca045f36e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"087d72f1-6bbe-44a3-82c1-1460a8ef4ebf","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:02Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:02Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:02Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:04Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"511797be-19ac-420d-80be-75d5ea245db7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7"} {"level":"info","ts":"2026-09-11T12:38:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:05Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"829746aa-2bdd-41ed-958a-ab390f8242d7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:06Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6b49a89b-3291-446b-80af-e544235ec91c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:06Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:07Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4886d8e7-c93f-48ca-9837-d6b81202b91a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:08Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c2e0c23a-9633-4b64-b9af-8cc022b456e8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:08Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:08Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"96bcfe67-4cf0-4985-83d3-0e3262d69d77","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:08Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"716a699f-5a0a-4da4-be6c-4b56ec0dfe84","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bd3bade9-94fb-4ff2-a4ae-31561485d323","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2da2c044-361d-4bfe-85be-ec23d47c61c7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4eea0318-493c-407f-a2ed-2e3e392da918","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:14Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7"} {"level":"info","ts":"2026-09-11T12:38:14Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7"} {"level":"info","ts":"2026-09-11T12:38:14Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:14Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:14Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:14Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:14Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d7b5e4c0-8c6f-40b8-97fe-fc3888247b7a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd"} {"level":"info","ts":"2026-09-11T12:38:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"209712f1-09b1-4df7-9415-b20f1d7352bd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:16Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:16Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:16Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2dc59b19-9730-408e-ab74-562e989260e7","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:16Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55"} {"level":"info","ts":"2026-09-11T12:38:16Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55"} {"level":"info","ts":"2026-09-11T12:38:16Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:16Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:18Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"eeb69caa-114f-4385-adfc-a7efd3685f55","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:19Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87"} {"level":"info","ts":"2026-09-11T12:38:19Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87"} {"level":"info","ts":"2026-09-11T12:38:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"c007c8a1-6fbc-433a-b151-06b68a0dce87","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2acb18a8-8231-4979-ab2f-46ef3c542ffb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2bf4e4b2-ab59-499b-bfa8-eca8e68e06b3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:22Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6a8dc8b5-6b19-4680-a923-6ace0d54e1bb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2a8b802d-f54b-4c49-b05a-1166d5bbe2b3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:23Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:23Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:23Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:23Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:23Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:23Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:24Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"50a25323-5b00-48d5-863c-d8e2fb13d107","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:24Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1"} {"level":"info","ts":"2026-09-11T12:38:24Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1"} {"level":"info","ts":"2026-09-11T12:38:24Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:24Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:24Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:24Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:24Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:25Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"70014a5d-b802-461d-8483-44277c5f7378","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:25Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:25Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:25Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:25Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:25Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:25Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:25Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"427f9f7d-2995-491d-a300-fd8ce3b9bda1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:25Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:26Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"91e5e20e-56a9-4fed-b9b1-db8f3a7a2cc1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:26Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735"} {"level":"info","ts":"2026-09-11T12:38:26Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735"} {"level":"info","ts":"2026-09-11T12:38:26Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:26Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:26Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:26Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:26Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c50d8e36-fdd5-428d-b95b-84972138c5d0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f98b0353-c672-4243-93a0-b3aec122f9d1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:28Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:28Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:28Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"99ade15c-16fa-4eed-9761-7d153f85e735","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c894fd66-fc5d-41e7-ae70-c393c2d72a3d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516"} {"level":"info","ts":"2026-09-11T12:38:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"875ea832-04d4-4069-b570-882d6a3be516","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a1bb2ec1-2e4b-4b4f-a149-c61d4d79bea3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"597fbce1-b6c3-4c49-a61c-dd4ae7bf1e33","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"61d4e24b-4922-4a06-a154-ed73498777ab","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:32Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:32Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:33Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d0d35256-cef4-4aec-ad6d-646d1507f51f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:33Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fe709379-274d-46b4-bc51-3ef746595e19","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:33Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"df6e48bd-156e-4c60-8ec8-32d073eeb62a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"reconcile","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"bd572617-315f-45f1-b5b4-02578b20cb55"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f2588511-47cb-4d8d-b163-f1529597aff4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:34Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"reconcile","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"1cc484f5-71c5-4bed-be35-d1c9833655ab"} {"level":"info","ts":"2026-09-11T12:38:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"reconcile","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"1531111b-1e2a-41cb-8884-4a8fa0889f41"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"23d9ddb3-d78e-47c9-a620-c2d29d0d5662","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7ce53d25-f53f-40b9-a042-cc8a5008dbe3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"15beb9f8-ee73-4f9a-af1d-01ff59538b7e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:37Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:40Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"d849b03d-cec0-4b56-9feb-7707eb6e56fb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26"} {"level":"info","ts":"2026-09-11T12:38:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c951773e-5a56-4668-9a80-415c75864c26","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"246d93fb-b23c-4d98-a8d4-da41176d36ff","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:42Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:42Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c9cfb005-83da-4002-be48-716c7bb8c96b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e9565c41-a382-4dd5-ab75-d0914986674a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:42Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:42Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bd704f96-0115-4375-b73a-b83dc6841040","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"06a191f8-1325-4235-9f9e-7a7b1b0d6ee8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:43Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"5dea0199-941c-4b5d-9b39-8b30de5dd3b9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b0e244cb-6157-4846-bdf6-6931652ceaab","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29129e32-2b41-45c6-87f8-88011f82e344","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"12cd1916-60a4-44da-ad87-4edc4c15b826","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f3164eb3-fc48-4366-ae64-2e2e3b2117e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"2a852e38-34c1-45db-b252-516222df208b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:46Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:46Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"17d92829-ebe5-4ef0-bad2-5738b634508c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c80dfb2-3311-4c1e-95cf-8b03f2f96e55","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:46Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:46Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"09cf568c-642c-4950-b4a4-f1ae4d75975f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d662e801-ce74-42c4-8709-8944845262e2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"05f96b02-c1b3-44dd-bbea-9a9559cae593","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:48Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:49Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:49Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"76244216-69ff-46b2-a5df-5895e58a5f43","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d"} {"level":"info","ts":"2026-09-11T12:38:52Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:52Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920dbac9-2621-4c27-ac23-444c8aef198d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"db94c352-513b-4463-b351-49dd0207fa58","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:52Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:52Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"66c66e89-9fb9-42e3-bf98-914adc299f1c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:53Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a8ad2ef9-92fc-4dc2-b125-2353114608e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:54Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:54Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8352ea6b-2370-40b5-b1a6-fc52f40ab5e2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:54Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f9c5bac9-1d5a-440b-adf9-2f7079e18d13","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:55Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:55Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"1dfc7427-470d-4625-a1d3-b2898c072881","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:38:56Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:56Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:38:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:57Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"ogx"} {"level":"info","ts":"2026-09-11T12:38:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:57Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c616c7e4-5f6a-4d3d-9680-2ddc0cb0430b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:57Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eceba85d-5ec2-40bc-bf1b-8b1a24abca0c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:57Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:38:57Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:38:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"trainer"} {"level":"info","ts":"2026-09-11T12:38:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:38:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:38:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:38:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"a072075f-5e09-410f-b47a-af55d0e6d6d4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:38:59Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cb126296-5c15-403f-af3c-837cc27c0f59","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5"} {"level":"info","ts":"2026-09-11T12:39:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"ogx"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:39:01Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"792a6d67-1425-4cd6-983e-6d769264c9a5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"078c5c68-3c19-42b7-b608-73b62d3af907","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:01Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:02Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7513f54d-3f39-441c-892b-0a4b4e3037e9","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fb7cd489-aacb-44e2-aa61-d54a95350549","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:02Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"trainer"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"152df32d-15bf-45a7-a979-698cbac3c5c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:39:03Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:39:03Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"ogx"} {"level":"info","ts":"2026-09-11T12:39:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:39:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:39:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"trainer"} {"level":"info","ts":"2026-09-11T12:39:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:39:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:39:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:05Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"ed79097d-9b72-45b4-809a-62f5b81b9e62","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b"} {"level":"info","ts":"2026-09-11T12:39:06Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:39:06Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:39:06Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"ogx"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"74402cd8-0f1d-4056-bdfd-4e8075b4ee8b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1a916ef6-0a26-413c-9b14-14947cb4308a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:06Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9063deb5-6dab-497c-b3c3-968ca785e83a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:39:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"trainer"} {"level":"info","ts":"2026-09-11T12:39:07Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:39:08Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:08Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:08Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:08Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:08Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:39:08Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"168a2ed4-3bae-43fd-a3f7-3c4eb3e09e9c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf"} {"level":"info","ts":"2026-09-11T12:39:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:09Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"ogx"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4845be85-a723-4d63-b0ac-76f8d7702edf","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:10Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:39:10Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7b87ec19-a04f-499d-8c54-02f00d983d56","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:10Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:39:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"trainer"} {"level":"info","ts":"2026-09-11T12:39:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:39:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b53333d9-f8ff-4f49-a66c-b7a8f8a445f3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4970867e-b179-4cdd-bd82-83183cb78b14","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:11Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"078aa5a6-fa02-4f80-bb60-84f84dbc5484","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d58076e-618a-40ae-8ce6-0bbeb3c311b6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"ogx"} {"level":"info","ts":"2026-09-11T12:39:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:39:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:39:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"trainer"} {"level":"info","ts":"2026-09-11T12:39:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:39:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:39:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:13Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:14Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"f29ccd04-60f0-4c39-b53d-fae8f607d5df","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"ogx"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"79c53c89-032b-4183-94ac-01b413561d8e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:18Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:39:18Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c43e8015-c57b-4711-ac1a-577325a8db12","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:18Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:39:18Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"trainer"} {"level":"info","ts":"2026-09-11T12:39:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:39:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:39:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:19Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"4d12076e-c8f0-4975-9c6c-d18e397524ed","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6"} {"level":"info","ts":"2026-09-11T12:39:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:39:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"ogx"} {"level":"info","ts":"2026-09-11T12:39:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"632b37f5-fba5-4b65-b4ba-d6b1461c1ff6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"03f44a95-0631-4b65-b6b7-bb33f5843d0b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:39:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"38432e2c-ff64-4e4a-ab26-061d10302cce","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:39:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"trainer"} {"level":"info","ts":"2026-09-11T12:39:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:39:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:39:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"187ba028-a02b-46ea-9cd5-3de2b47a5e4c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"ogx"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b9fc60f5-411b-415a-a9a3-a8b73d8c0e24","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:28Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:39:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"fbbe859f-a014-404c-a287-c685baa7953b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:28Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:39:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"trainer"} {"level":"info","ts":"2026-09-11T12:39:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:39:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:39:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:29Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"6e456bc7-800e-4181-abcc-ee97d62006b4","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150"} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150"} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"reconcile","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6"} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"apply","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6"} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.enableModulesFromPlatform"} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-09-11T12:39:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"sparkoperator"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"ogx"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1c472b33-5988-4031-afe0-8549539ae150","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.syncPlatformCR"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledComponents"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.cleanupDisabledModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionModuleCRs"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"8266e1ed-5b5c-4801-b8e5-a10791049ffa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"mlflowoperator"} {"level":"info","ts":"2026-09-11T12:39:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"aigateway"} {"level":"info","ts":"2026-09-11T12:39:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"trainer"} {"level":"info","ts":"2026-09-11T12:39:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"mcplifecycleoperator"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-09-11T12:39:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"dashboard","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"modelregistry","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"monitoring","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"workbenches","runlevel":"20"} {"level":"info","ts":"2026-09-11T12:39:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"kserve","runlevel":"31"} {"level":"info","ts":"2026-09-11T12:39:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules","msg":"provisioning module operator","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","module":"feastoperator","runlevel":"32"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/provision.ExtractUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.checkUpgradeGates"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectPlatformConfig"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:39:38Z","msg":"Executing action","controller":"modules","controllerGroup":"config.opendatahub.io","controllerKind":"Platform","Platform":{"name":"default"},"namespace":"","name":"default","reconcileID":"afbc56d5-92f3-4ef3-af24-8c442487a7d6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:43:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:43:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a09c48d7-1199-4229-ac1d-a03220e1ace2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:52:02Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:52:02Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"EnvoyFilter exposes internal implementation details that may change at any time. Prefer other APIs if possible, and exercise extreme caution, especially around upgrades.","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5a2c0adc-e1e4-4b3d-aad3-5902b4b7b517","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:52:02Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:52:02Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"87f18416-5c6e-496e-8ad3-988480dab62c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:52:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:52:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"48981ab8-e1d7-4308-b4f6-9436c798f990","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:52:31Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"openldap","reconcileID":"2aba9aee-763c-46f5-bae6-4bb0d9e63457"} {"level":"info","ts":"2026-09-11T12:52:31Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"openldap","reconcileID":"b9145a65-2f9f-41b6-9603-ef6d9463a2ee"} {"level":"info","ts":"2026-09-11T12:52:31Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"openldap","reconcileID":"a43fe896-e4f4-40fc-863e-aff5b50e17d3"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-r7tdx","reconcileID":"54e8c854-6f19-4bf4-9988-dafa6dca5dff"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-r7tdx","reconcileID":"4e30d1d7-734e-4fa6-85fa-3432d712a74f"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-r7tdx","reconcileID":"2b7f2fa7-2c2b-45f8-b4c2-869294bdbab4"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:58:22Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9270765-6db0-49a1-bd7a-39a3306231fa","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T12:58:22Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T12:58:23Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T12:58:23Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T12:58:23Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T12:58:23Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T12:58:23Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T12:58:23Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:58:23Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:58:23Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T12:58:23Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T12:58:23Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6e034966-29c2-4a3f-a985-73cb3ba00964","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:02:42Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-kppl8","reconcileID":"71b44a84-2fd0-459a-b843-360b8d71093b"} {"level":"info","ts":"2026-09-11T13:02:42Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-kppl8","reconcileID":"554bbcd4-de83-4f2c-a41e-4641793006e5"} {"level":"info","ts":"2026-09-11T13:02:42Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-kppl8","reconcileID":"5950f9e5-789d-4762-8a2e-370be36bbf99"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:02:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:43Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"EnvoyFilter exposes internal implementation details that may change at any time. Prefer other APIs if possible, and exercise extreme caution, especially around upgrades.","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6525b35-7cb1-4ac5-864c-cc2b639597a3","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:02:43Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:43Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"06e3755c-0057-4726-bfa8-c697805b6eac","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:02:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b7e4ba1d-1cff-495e-a751-790e3404ec1c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:02:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"96e3c6c3-4044-4a25-b739-a7086e3f2fff","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:02:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:02:53Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ccf5426d-17f1-4493-8706-e33ff58bffdb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:03:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"04bef096-ef0d-4d1b-ac56-9112a7cf9767","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:03:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"476ad894-fee5-42c9-a4ce-ece223597e8f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:03:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:03:29Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"35c823ef-78ba-4d08-ab86-d3b550d3dcc5","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:04:40Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"EnvoyFilter exposes internal implementation details that may change at any time. Prefer other APIs if possible, and exercise extreme caution, especially around upgrades.","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"69140b66-6e4e-49b8-8224-a74917616acd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:04:40Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67ddeea9-bf2c-455d-b901-ff82bf6af5ce","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:04:40Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"EnvoyFilter exposes internal implementation details that may change at any time. Prefer other APIs if possible, and exercise extreme caution, especially around upgrades.","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a3e82af-6b7f-40cf-b59a-b26599c2fd93","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:04:40Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:04:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"fdf11fff-8439-462f-b9bb-bf03e7819dd2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:05:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:05:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c3eb5ccc-056f-4279-aa0c-63995274e89a","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:05:32Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cypress-user-perm-test-526006","reconcileID":"13acdc48-b794-47e7-a266-18b3d69dd71b"} {"level":"info","ts":"2026-09-11T13:05:32Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cypress-user-perm-test-526006","reconcileID":"8e684a9b-e2ef-4315-b7f3-e1615884ea3d"} {"level":"info","ts":"2026-09-11T13:05:32Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cypress-user-perm-test-526006","reconcileID":"9174f555-38e7-4554-90b0-b61c755e81f4"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:05:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:05:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"27223d4d-cda7-4410-a709-0c4a792f70c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:05:49Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:05:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"137163d6-2b2b-4013-b1a4-ce65ce07985e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:06:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b54c48fb-d060-47c6-8d63-6c8927bd501d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:06:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"2a08fada-e61c-477e-bc11-a9add3a67404","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:06:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b8f7c967-353e-4140-a8de-2ee0add69113","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:06:49Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:06:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"876de940-0afb-4ed1-affa-bd4d7b873d82","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:06:57Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cypress-edit-test-project-609558","reconcileID":"dc593f25-071f-4632-a89d-cd1caa7760bf"} {"level":"info","ts":"2026-09-11T13:06:57Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cypress-edit-test-project-609558","reconcileID":"92c9afc9-adb8-408d-be5b-9799931ff951"} {"level":"info","ts":"2026-09-11T13:06:57Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cypress-edit-test-project-609558","reconcileID":"147f7c0f-1c88-43b3-8ed0-746431acbd5b"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:07:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"329335ad-5cde-4cf9-8207-aa95067ca812","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:07:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e2f9124-76f8-4a15-9f61-110e26439eae","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:07:23Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"qe-cluster-storage-sc-dsp","reconcileID":"87db4a42-64f6-4582-b8fc-ec08fb1f56fd"} {"level":"info","ts":"2026-09-11T13:07:23Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"qe-cluster-storage-sc-dsp","reconcileID":"cde8bd27-9a63-4882-928f-38195f41745b"} {"level":"info","ts":"2026-09-11T13:07:23Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"qe-cluster-storage-sc-dsp","reconcileID":"5465d894-311a-40f5-afcd-e16ae26f022b"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:07:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8f955460-179f-41bb-a07a-915af8ea62c1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:07:49Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:07:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"79fdb0bd-b6f1-4b45-bba1-c4770cdaa3f8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:08:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1ab00f26-0bcf-4cb4-b25d-7dd6cfd25eeb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:08:09Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-cluster-storage-access-modes","reconcileID":"3254090b-b3d7-47e1-ac40-d2dce914acf8"} {"level":"info","ts":"2026-09-11T13:08:09Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-cluster-storage-access-modes","reconcileID":"d714a011-ebba-46a6-a728-29cb8460d3d5"} {"level":"info","ts":"2026-09-11T13:08:09Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-cluster-storage-access-modes","reconcileID":"cfb6e569-6c7f-49cb-9327-d400d94bb22a"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:08:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c848390f-05f5-4b75-ba93-4d6b32e08c24","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:08:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c5b3d45-fa3b-421c-af00-8a24739d40c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:08:49Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:08:49Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b86b27b8-53c7-40a5-accb-ae956fb5b6ee","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:08:59Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cypress-test-connection-type-185825","reconcileID":"50626c7a-125f-4920-b3fb-a65905341bc4"} {"level":"info","ts":"2026-09-11T13:08:59Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cypress-test-connection-type-185825","reconcileID":"c1ea9e82-862d-4e6d-bf8d-1ca15bde334c"} {"level":"info","ts":"2026-09-11T13:08:59Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"cypress-test-connection-type-185825","reconcileID":"54b34411-599e-42d0-bb6a-d0e47202842b"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:09:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c14c2384-3e80-4cb1-a5ac-015901b78ad6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:09:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0160720f-cdb3-483e-b3f8-ea839637a6d1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:09:34Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:34Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d65d54f8-a683-4b84-97bf-183a7cb384d2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:09:38Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-workbench-storage-preset-167984","reconcileID":"dac78089-9ede-4988-a128-4320c9af4589"} {"level":"info","ts":"2026-09-11T13:09:38Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-workbench-storage-preset-167984","reconcileID":"c5b43192-1f6f-43b8-a02e-f0a0a1ee2cee"} {"level":"info","ts":"2026-09-11T13:09:38Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-workbench-storage-preset-167984","reconcileID":"44125f63-0b56-4062-978b-896831ad628c"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:09:56Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1c21ca3a-a08b-4ce7-9e41-18207bfbaa0c","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:09:56Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:09:56Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd4044b2-e437-4230-87b7-c2a8d50b2a94","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:10:04Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd"} {"level":"info","ts":"2026-09-11T13:10:04Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd"} {"level":"info","ts":"2026-09-11T13:10:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:04Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:10:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:10:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:10:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:10:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:10:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:10:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1cb3dd4f-0d77-467b-87c1-13c151076cbd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:10:19Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1"} {"level":"info","ts":"2026-09-11T13:10:19Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1"} {"level":"info","ts":"2026-09-11T13:10:19Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:10:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"20b1b825-0373-4630-99d0-ec10411a0fc1","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:10:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b5f7566d-c82e-4154-921d-28c17aa7d394","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:10:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8709de11-eb0a-43b1-ac08-0d15954f656e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:10:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c79cb41f-2aab-4f40-9eb6-f4e981d0bd42","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:10:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"322cddad-a25f-45b3-992a-f2fc0f002e3d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:10:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:10:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5213ac31-7137-40e9-965f-547bbba12e9e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:11:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:11:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"aacb7a6e-b38a-433a-9e9f-34edf45d83f0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:11:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:11:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"6024ace6-89f5-4c98-a477-2ceb01b242c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:11:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:11:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"b32ac0c6-a884-43bc-ad95-7838094f78da","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:13:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"78f3852a-2271-49a6-a403-4954e3e44429","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:13:11Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:11Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bdb3bc1c-95ec-49e7-90b7-2fd990bcf625","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:13:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40162038-0474-4402-a219-302fce387198","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:13:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9da098b5-b913-4a65-9675-8dd7e2773beb","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:13:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bf6e1d9c-955e-4083-8ebf-f18330f4ebc0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:13:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:39Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f9448b13-9c51-4b62-8136-4b90e30c9445","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:13:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"888657bd-cf66-4a8d-8341-4c6f6324cb25","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:13:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:13:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3694ab3c-1845-40e9-84fd-9c3a71f9aedd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:14:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"26b468f1-3ab6-4904-8402-df0a45444da6","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:14:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"63095ff1-9a19-41de-926e-26403ef66c4e","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:14:13Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:13Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40046b02-423a-4be1-8104-6dd3a1a7e7c2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:14:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c6d8f5b0-3030-4d3b-b877-fbe0d0d6a73d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:14:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0223477e-4f04-4fc4-a514-42f48f812853","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:14:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0a236d7-805e-4a2b-a907-8dcad1cd8b11","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:14:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3d10f50a-49b0-4a37-8c7c-4cb28b75fbf2","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:14:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:50Z","logger":"action.github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm","msg":"EnvoyFilter exposes internal implementation details that may change at any time. Prefer other APIs if possible, and exercise extreme caution, especially around upgrades.","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:14:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"acc39bb8-ca3e-46ac-ab99-ef8996b708c0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:15:16Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0"} {"level":"info","ts":"2026-09-11T13:15:16Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0"} {"level":"info","ts":"2026-09-11T13:15:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:15:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:15:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"67dd5ec2-b609-4499-8cb7-d9337ff303e0","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:15:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:15:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0c42d6a3-5b4b-44c3-b562-3c9509f72aa8","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:16:05Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:05Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"3ed693c3-448d-46ad-b2b8-3674b34e1369","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:16:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"037fb6db-d206-4488-9b56-af2958b3eccd","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:16:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"0ebea2a3-2f1b-455f-a287-83746b755e38","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:16:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:16:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:16:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"e6991046-d147-40f3-b959-5607aacfcf1d","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:16:46Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-register-store-662874","reconcileID":"96129d9d-00fc-4017-817e-b9f2aedf3201"} {"level":"info","ts":"2026-09-11T13:16:46Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-register-store-662874","reconcileID":"c16739cc-5b11-4a62-8421-ceeaa6748398"} {"level":"info","ts":"2026-09-11T13:16:47Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-register-store-662874","reconcileID":"74e4d8d0-7e23-4595-b593-40eaffbacb33"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:17:06Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:06Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"079d1a1d-923c-498d-9057-9dc0c546c168","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:17:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30747de1-5002-447f-972a-3fa372ba6c96","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:17:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4f46f471-ad29-4fc8-805e-52c446ed884b","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:17:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"90d041dd-cd99-45c7-a710-3dfec9b1a888","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:17:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"32662632-68ce-402b-a8af-3c1e668cd5ad","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction"} {"level":"info","ts":"2026-09-11T13:17:51Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirectsAction.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-09-11T13:17:51Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"c995d0cf-4710-4571-929c-7a6e3d81e41f","action":"github.com/opendatahub-io/odh-platform-utilities/framework/controller/reconciler.(*dynamicWatchAction).run-fm"} ==================== OPERATOR POD LOG: rhods-operator-5c4c6d8cfd-z6kcv.log ==================== {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"Cluster config","Operator Namespace":"redhat-ods-operator","Application Namespace":"redhat-ods-applications","Release":{"name":"OpenShift AI Self-Managed","version":"3.6.0-ea.1"},"Cluster":{"type":"OpenShift","version":"4.21.31"}} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"datasciencecluster.opendatahub.io/v1, Kind=DataScienceCluster"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"dscinitialization.opendatahub.io/v1, Kind=DSCInitialization"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-hardware-profile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-prometheus-monitors"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-acceleratorprofile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-hardwareprofile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"service","name":"certconfigmapgenerator"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"service","name":"gateway"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"service","name":"setupcontroller"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"service","name":"auth"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"component","name":"datasciencepipelines"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"component","name":"ray"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"component","name":"kueue"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"component","name":"trustyai"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"starting manager"} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/etc/metrics-certs/tls.crt","key":"/etc/metrics-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/etc/metrics-certs/tls.crt","key":"/etc/metrics-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"Starting EventSource","controller":"tlssecurityprofilewatcher","source":"kind source: *v1.APIServer"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"Attempting to acquire leader lease...","lock":"redhat-ods-operator/07ed84f7.opendatahub.io"} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"Starting Controller","controller":"tlssecurityprofilewatcher"} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"Starting workers","controller":"tlssecurityprofilewatcher","worker count":1} ==================== OPERATOR POD LOG: rhods-operator-5c4c6d8cfd-zrrwx.log ==================== {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"Cluster config","Operator Namespace":"redhat-ods-operator","Application Namespace":"redhat-ods-applications","Release":{"name":"OpenShift AI Self-Managed","version":"3.6.0-ea.1"},"Cluster":{"type":"OpenShift","version":"4.21.31"}} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"datasciencecluster.opendatahub.io/v1, Kind=DataScienceCluster"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-datasciencecluster-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"dscinitialization.opendatahub.io/v1, Kind=DSCInitialization"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v1"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dscinitialization-v2"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-hardware-profile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-prometheus-monitors"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-acceleratorprofile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-dashboard-hardwareprofile"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"service","name":"certconfigmapgenerator"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"service","name":"gateway"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"service","name":"setupcontroller"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"service","name":"auth"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"component","name":"datasciencepipelines"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"component","name":"ray"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"component","name":"kueue"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"creating reconciler","type":"component","name":"trustyai"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"starting manager"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/etc/metrics-certs/tls.crt","key":"/etc/metrics-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/etc/metrics-certs/tls.crt","key":"/etc/metrics-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"Starting EventSource","controller":"tlssecurityprofilewatcher","source":"kind source: *v1.APIServer"} {"level":"info","ts":"2026-09-11T12:37:14Z","logger":"setup","msg":"Attempting to acquire leader lease...","lock":"redhat-ods-operator/07ed84f7.opendatahub.io"} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"Starting Controller","controller":"tlssecurityprofilewatcher"} {"level":"info","ts":"2026-09-11T12:37:14Z","msg":"Starting workers","controller":"tlssecurityprofilewatcher","worker count":1} ==================== WORKLOAD POD LOG: redhat-ods-applications/agent-ops-ui-694ff95d86-wxdgd.log ==================== time=2026-09-11T12:38:36.884Z level=INFO msg="Added CA bundle" path=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem time=2026-09-11T12:38:36.884Z level=INFO msg="Added CA bundle" path=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt time=2026-09-11T12:38:36.884Z level=INFO msg="Added CA bundle" path=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt time=2026-09-11T12:38:36.992Z level=INFO msg="Added CA bundle" path=/etc/pki/tls/certs/odh-trusted-ca-bundle.crt time=2026-09-11T12:38:36.992Z level=INFO msg="Using real BFF client factory" time=2026-09-11T12:38:36.992Z level=INFO msg="Using Kubernetes agent data client" time=2026-09-11T12:38:36.992Z level=ERROR msg="failed to create OpenAPI handler; docs routes disabled" error="reading OpenAPI spec openapi/src/agent-ops.yaml: open openapi/src/agent-ops.yaml: no such file or directory" time=2026-09-11T12:38:36.993Z level=INFO msg="starting server" addr=:8843 "TLS enabled"=true ==================== WORKLOAD POD LOG: redhat-ods-applications/aihub-controller-manager-846f48f9b4-rzpmk.log ==================== 2026-09-11T12:37:53Z INFO setup cluster capabilities detected {"isOpenShift": true, "hasUserAPI": true, "hasConfigAPI": true, "hasAuthAPI": true} 2026-09-11T12:37:53Z INFO setup setting up aihub controller 2026-09-11T12:37:53Z INFO setup starting manager 2026-09-11T12:37:53Z INFO controller-runtime.metrics Starting metrics server 2026-09-11T12:37:53Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} I0911 12:37:53.814233 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="redhat-ods-applications/aihub-controller-manager" 2026-09-11T12:37:53Z INFO controller-runtime.certwatcher Updated current TLS certificate {"cert": "/tmp/k8s-metrics-server/serving-certs/tls.crt", "key": "/tmp/k8s-metrics-server/serving-certs/tls.key"} 2026-09-11T12:37:53Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8443", "secure": true} 2026-09-11T12:37:53Z INFO controller-runtime.certwatcher Starting certificate poll+watcher {"cert": "/tmp/k8s-metrics-server/serving-certs/tls.crt", "key": "/tmp/k8s-metrics-server/serving-certs/tls.key", "interval": "10s"} I0911 12:37:53.845332 1 leaderelection.go:272] "Successfully acquired lease" lock="redhat-ods-applications/aihub-controller-manager" 2026-09-11T12:37:53Z DEBUG events aihub-controller-manager-846f48f9b4-rzpmk_4995b198-955f-4c59-ad60-bee89cf62824 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"redhat-ods-applications","name":"aihub-controller-manager","uid":"0503cdd2-19a3-47ec-b645-18a26373cd3c","apiVersion":"coordination.k8s.io/v1","resourceVersion":"28162"}, "reason": "LeaderElection"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.ConfigMap"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.Deployment"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.Service"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.ServiceAccount"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.ConfigMap"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1alpha1.AIHub"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.Role"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.RoleBinding"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.ClusterRole"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.ClusterRoleBinding"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.ValidatingWebhookConfiguration"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1.MutatingWebhookConfiguration"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "source": "kind source: *v1alpha1.Catalog"} 2026-09-11T12:37:55Z INFO Starting Controller {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub"} 2026-09-11T12:37:55Z INFO Starting workers {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "worker count": 1} 2026-09-11T12:37:55Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "787c1492-4719-4324-85bd-3bac72f7ebb5", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:37:55Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "787c1492-4719-4324-85bd-3bac72f7ebb5", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:55Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "23e59d39-12e8-4cc5-96ac-c2cfb00cd3cf", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:37:56Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "23e59d39-12e8-4cc5-96ac-c2cfb00cd3cf", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:56Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "d898eed9-5d50-4180-8d94-4291f453ea2c", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:37:56Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "d898eed9-5d50-4180-8d94-4291f453ea2c", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:56Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "5df838b3-def9-4cc3-a822-ba54fe67eedf", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:37:56Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "5df838b3-def9-4cc3-a822-ba54fe67eedf", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:56Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "27abc7f5-c473-4b8b-909a-0f44da65de51", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:37:57Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "27abc7f5-c473-4b8b-909a-0f44da65de51", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:57Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "9fbdface-306e-4be5-ab28-ba09000572bd", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:37:57Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "9fbdface-306e-4be5-ab28-ba09000572bd", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:57Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "9048fdb0-9d17-4e5f-8f3a-0ee974aba268", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:37:58Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "9048fdb0-9d17-4e5f-8f3a-0ee974aba268", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:58Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "c92cd5d2-6dff-466f-85cc-e85c0a8764c6", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:37:58Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "c92cd5d2-6dff-466f-85cc-e85c0a8764c6", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:59Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "cddae3a3-e6c1-4e4f-b43f-2f98996a00cb", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:37:59Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "cddae3a3-e6c1-4e4f-b43f-2f98996a00cb", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:00Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "21b9b69b-a5ae-4c3e-a8c6-d9a57fe7b13f", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:01Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "21b9b69b-a5ae-4c3e-a8c6-d9a57fe7b13f", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:03Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "f6ce15e2-8948-4980-a584-7d8d9007f941", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:04Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "f6ce15e2-8948-4980-a584-7d8d9007f941", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:09Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "05c883ba-fab6-4d22-a106-0d6e7a71a69e", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:09Z ERROR Reconciler error {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "05c883ba-fab6-4d22-a106-0d6e7a71a69e", "error": "deploying child operator resources: failure deploying resource redhat-ods-applications/catalog-controller-manager: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/catalog-controller-manager: serviceaccounts \"catalog-controller-manager\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind AIHub: no matches for kind \"AIHub\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:19Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "dda5843f-a11e-4193-b7f9-eb644b5ff5e6", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:21Z INFO child deployment not yet Available, requeuing {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "dda5843f-a11e-4193-b7f9-eb644b5ff5e6", "deployment": "model-registry-operator-controller-manager"} 2026-09-11T12:38:22Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "e2c62197-58f2-49e9-b9b8-0e1fe3968d3a", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:22Z INFO child deployment not yet Available, requeuing {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "e2c62197-58f2-49e9-b9b8-0e1fe3968d3a", "deployment": "model-registry-operator-controller-manager"} 2026-09-11T12:38:22Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "8936489f-b6e5-43cb-b932-5192fed4d209", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:23Z INFO child deployment not yet Available, requeuing {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "8936489f-b6e5-43cb-b932-5192fed4d209", "deployment": "model-registry-operator-controller-manager"} 2026-09-11T12:38:23Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "f15dc194-6951-4a46-a24c-1e8df443f75d", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:24Z INFO child deployment not yet Available, requeuing {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "f15dc194-6951-4a46-a24c-1e8df443f75d", "deployment": "model-registry-operator-controller-manager"} 2026-09-11T12:38:34Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "5b550656-7f94-4063-8206-456273b0786c", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:35Z INFO child deployment not yet Available, requeuing {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "5b550656-7f94-4063-8206-456273b0786c", "deployment": "catalog-controller-manager"} 2026-09-11T12:38:35Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "6dec7c7f-e36e-4c1d-8088-4f46aab3f299", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:36Z INFO child deployment not yet Available, requeuing {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "6dec7c7f-e36e-4c1d-8088-4f46aab3f299", "deployment": "catalog-controller-manager"} 2026-09-11T12:38:43Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "6b2973fc-69bf-43a6-91f4-522e571ddaad", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:44Z INFO creating {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "6b2973fc-69bf-43a6-91f4-522e571ddaad", "kind": "aihub.opendatahub.io/v1alpha1, Kind=Catalog", "name": "catalog"} 2026-09-11T12:38:44Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "6b2973fc-69bf-43a6-91f4-522e571ddaad"} 2026-09-11T12:38:44Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "6075cc71-20e1-4a40-a9bb-631fc5bb5a73", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:45Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "6075cc71-20e1-4a40-a9bb-631fc5bb5a73"} 2026-09-11T12:38:46Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "ea3d0472-adfc-4fe0-85b5-8386196f2639", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:46Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "ea3d0472-adfc-4fe0-85b5-8386196f2639"} 2026-09-11T12:38:57Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "25f32c18-8ca4-4210-8ab3-e6bdce912e62", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:38:57Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "25f32c18-8ca4-4210-8ab3-e6bdce912e62"} 2026-09-11T12:39:19Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "db05b1ef-2085-4acc-98d2-134dc9cc04f1", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:39:20Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "db05b1ef-2085-4acc-98d2-134dc9cc04f1"} 2026-09-11T12:39:24Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "51d287d0-9ecb-4688-9b29-eaa056b5f95b", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T12:39:24Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "51d287d0-9ecb-4688-9b29-eaa056b5f95b"} 2026-09-11T13:11:47Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "2285fdcd-74da-4f54-af9f-eab0cba7a1f9", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T13:11:49Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "2285fdcd-74da-4f54-af9f-eab0cba7a1f9"} 2026-09-11T13:11:49Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "03a8e455-70e8-4d31-9cbc-67657fe19e56", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T13:11:50Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "03a8e455-70e8-4d31-9cbc-67657fe19e56"} 2026-09-11T13:12:00Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "7b748811-8680-41bc-a37a-9ae3d880bf45", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T13:12:01Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "7b748811-8680-41bc-a37a-9ae3d880bf45"} 2026-09-11T13:12:01Z INFO reconciling AIHub {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "1df359bc-a542-4540-8372-0a292b63303e", "applicationNamespace": "redhat-ods-applications", "instancesNamespace": "rhoai-model-registries"} 2026-09-11T13:12:01Z INFO AIHub reconciliation complete {"controller": "aihub", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "AIHub", "AIHub": {"name":"default-aihub"}, "namespace": "", "name": "default-aihub", "reconcileID": "1df359bc-a542-4540-8372-0a292b63303e"} ==================== WORKLOAD POD LOG: redhat-ods-applications/automl-ui-74d9b97d49-25jfx.log ==================== time=2026-09-11T12:38:42.385Z level=INFO msg="Added CA bundle" path=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem time=2026-09-11T12:38:42.385Z level=INFO msg="Added CA bundle" path=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt time=2026-09-11T12:38:42.385Z level=INFO msg="Added CA bundle" path=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt time=2026-09-11T12:38:42.482Z level=INFO msg="Added CA bundle" path=/etc/pki/tls/certs/odh-trusted-ca-bundle.crt time=2026-09-11T12:38:42.483Z level=INFO msg="starting server" addr=:8643 "TLS enabled"=true ==================== WORKLOAD POD LOG: redhat-ods-applications/autorag-ui-58f548cbd-8h2dd.log ==================== time=2026-09-11T12:38:42.388Z level=INFO msg="Added CA bundle" path=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem time=2026-09-11T12:38:42.389Z level=INFO msg="Added CA bundle" path=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt time=2026-09-11T12:38:42.389Z level=INFO msg="Added CA bundle" path=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt time=2026-09-11T12:38:42.485Z level=INFO msg="Added CA bundle" path=/etc/pki/tls/certs/odh-trusted-ca-bundle.crt time=2026-09-11T12:38:42.486Z level=INFO msg="starting server" addr=:8743 "TLS enabled"=true ==================== WORKLOAD POD LOG: redhat-ods-applications/catalog-controller-manager-66fc49dffb-v5mww.log ==================== 2026-09-11T12:38:22Z INFO setup cluster capabilities detected {"isOpenShift": true, "hasUserAPI": true, "hasConfigAPI": true, "hasAuthAPI": true} 2026-09-11T12:38:22Z INFO setup default catalog config {"REGISTRIES_NAMESPACE": "rhoai-model-registries", "DEFAULT_DOMAIN": ""} 2026-09-11T12:38:22Z INFO setup parsed kubernetes templates {"templates": "; defined templates are: \"http-route.yaml.tmpl\", \"postgres-pvc.yaml.tmpl\", \"kube-rbac-proxy-network-policy.yaml.tmpl\", \"catalog-agent-configmap.yaml.tmpl\", \"catalog-kube-rbac-proxy-network-policy.yaml.tmpl\", \"catalog-postgres-deployment.yaml.tmpl\", \"gateway-httproute.yaml.tmpl\", \"deployment.yaml.tmpl\", \"postgres-deployment.yaml.tmpl\", \"service.yaml.tmpl\", \"kube-rbac-proxy-https-route.yaml.tmpl\", \"kube-rbac-proxy-role-binding.yaml.tmpl\", \"catalog-role.yaml.tmpl\", \"kube-rbac-proxy-config.yaml.tmpl\", \"catalog-kube-rbac-proxy-role-binding.yaml.tmpl\", \"catalog-postgres-network-policy.yaml.tmpl\", \"serviceaccount.yaml.tmpl\", \"catalog-default-configmap.yaml.tmpl\", \"catalog-gateway-httproute.yaml.tmpl\", \"catalog-kube-rbac-proxy-config.yaml.tmpl\", \"catalog-kube-rbac-proxy-https-route.yaml.tmpl\", \"catalog-serviceaccount.yaml.tmpl\", \"postgres-network-policy.yaml.tmpl\", \"catalog-deployment.yaml.tmpl\", \"catalog-mcp-configmap.yaml.tmpl\", \"catalog-rolebinding.yaml.tmpl\", \"catalog-service.yaml.tmpl\", \"postgres-service.yaml.tmpl\", \"catalog-postgres-secret.yaml.tmpl\", \"group.yaml.tmpl\", \"catalog-admin-rolebinding.yaml.tmpl\", \"catalog-configmap.yaml.tmpl\", \"catalog-postgres-service.yaml.tmpl\", \"gateway-reference-grant.yaml.tmpl\", \"role-binding.yaml.tmpl\", \"role.yaml.tmpl\", \"catalog-admin-role.yaml.tmpl\""} 2026-09-11T12:38:22Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-aihub-opendatahub-io-catalog"} 2026-09-11T12:38:22Z INFO setup starting catalog manager 2026-09-11T12:38:22Z INFO controller-runtime.metrics Starting metrics server 2026-09-11T12:38:22Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} 2026-09-11T12:38:22Z INFO controller-runtime.webhook Starting webhook server 2026-09-11T12:38:22Z INFO controller-runtime.certwatcher Updated current TLS certificate {"cert": "/tmp/k8s-webhook-server/serving-certs/tls.crt", "key": "/tmp/k8s-webhook-server/serving-certs/tls.key"} 2026-09-11T12:38:22Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443} 2026-09-11T12:38:22Z INFO controller-runtime.certwatcher Starting certificate poll+watcher {"cert": "/tmp/k8s-webhook-server/serving-certs/tls.crt", "key": "/tmp/k8s-webhook-server/serving-certs/tls.key", "interval": "10s"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "tlssecurityprofilewatcher", "source": "kind source: *v1.APIServer"} I0911 12:38:22.690926 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="redhat-ods-applications/catalog.opendatahub.io" I0911 12:38:22.720429 1 leaderelection.go:272] "Successfully acquired lease" lock="redhat-ods-applications/catalog.opendatahub.io" 2026-09-11T12:38:22Z DEBUG events catalog-controller-manager-66fc49dffb-v5mww_7ebe9101-deb8-4971-ade0-02e3458e3346 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"redhat-ods-applications","name":"catalog.opendatahub.io","uid":"f7a78c15-c006-4789-85d3-cc7c72a18489","apiVersion":"coordination.k8s.io/v1","resourceVersion":"29037"}, "reason": "LeaderElection"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *unstructured.Unstructured[services.platform.opendatahub.io/v1alpha1 Auth]"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1alpha1.Catalog"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.Deployment"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.Service"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.ServiceAccount"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.ConfigMap"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.Secret"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.NetworkPolicy"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.Role"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.RoleBinding"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.Route"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.ConfigMap"} 2026-09-11T12:38:22Z INFO Starting EventSource {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "source": "kind source: *v1.ClusterRoleBinding"} 2026-09-11T12:38:22Z INFO Starting Controller {"controller": "tlssecurityprofilewatcher"} 2026-09-11T12:38:22Z INFO Starting workers {"controller": "tlssecurityprofilewatcher", "worker count": 1} 2026-09-11T12:38:22Z DEBUG Reconciling APIServer TLS profile {"controller": "tlssecurityprofilewatcher", "reconcileID": "cd951a78-d284-4d32-a4fb-230225874917", "name": "cluster"} 2026-09-11T12:38:22Z DEBUG Finished reconciling APIServer TLS profile {"controller": "tlssecurityprofilewatcher", "reconcileID": "cd951a78-d284-4d32-a4fb-230225874917", "name": "cluster"} 2026-09-11T12:38:22Z INFO Starting Controller {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog"} 2026-09-11T12:38:22Z INFO Starting workers {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "worker count": 1} 2026-09-11T12:38:23Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8443", "secure": true} 2026-09-11T12:38:44Z INFO admission validate create catalog {"object": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "resource": {"group":"aihub.opendatahub.io","version":"v1alpha1","resource":"catalogs"}, "user": "system:serviceaccount:redhat-ods-applications:aihub-controller-manager", "requestID": "ff01cdce-12b1-4673-b3ca-a426e3b196cb", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:44Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:44Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:44Z INFO Creating postgres secret with random password {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "secret": "model-catalog-postgres"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "/, Kind=", "name": "model-catalog-postgres"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "/v1, Kind=ServiceAccount", "name": "model-catalog"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "/v1, Kind=ConfigMap", "name": "default-catalog-sources"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "/v1, Kind=ConfigMap", "name": "model-catalog-sources"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "/v1, Kind=ConfigMap", "name": "mcp-catalog-sources"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "/v1, Kind=ConfigMap", "name": "agent-catalog-sources"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "apps/v1, Kind=Deployment", "name": "model-catalog"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "/v1, Kind=Service", "name": "model-catalog"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "rbac.authorization.k8s.io/v1, Kind=Role", "name": "model-catalog"} 2026-09-11T12:38:44Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "rbac.authorization.k8s.io/v1, Kind=RoleBinding", "name": "model-catalog-authenticated"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "rbac.authorization.k8s.io/v1, Kind=Role", "name": "model-catalog-admin"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "rbac.authorization.k8s.io/v1, Kind=RoleBinding", "name": "model-catalog-admin-binding"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "apps/v1, Kind=Deployment", "name": "model-catalog-postgres"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "/v1, Kind=Service", "name": "model-catalog-postgres"} 2026-09-11T12:38:45Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "networking.k8s.io/v1, Kind=NetworkPolicy", "name": "model-catalog-postgres"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "gateway.networking.k8s.io/v1beta1, Kind=ReferenceGrant", "name": "allow-gateway-httproutes"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "gateway.networking.k8s.io/v1, Kind=HTTPRoute", "name": "model-catalog"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "route.openshift.io/v1, Kind=Route", "name": "model-catalog-https"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "networking.k8s.io/v1, Kind=NetworkPolicy", "name": "model-catalog-https-route"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "/v1, Kind=ConfigMap", "name": "model-catalog-kube-rbac-proxy-config"} 2026-09-11T12:38:45Z INFO creating {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f8abab71-9289-4ed3-aed3-2fda0b690033", "kind": "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding", "name": "model-catalog-auth-delegator"} 2026-09-11T12:38:46Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "053d4bcf-8c6f-43b4-98f7-8f3c629dc813", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:46Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:46Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "053d4bcf-8c6f-43b4-98f7-8f3c629dc813", "secret": "model-catalog-postgres"} 2026-09-11T12:38:46Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "053d4bcf-8c6f-43b4-98f7-8f3c629dc813"} 2026-09-11T12:38:46Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "36dd79b8-f0f1-4559-8027-041136285ea5", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:46Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:46Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "36dd79b8-f0f1-4559-8027-041136285ea5", "secret": "model-catalog-postgres"} 2026-09-11T12:38:46Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "36dd79b8-f0f1-4559-8027-041136285ea5"} 2026-09-11T12:38:46Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "fa600fa5-0a94-414c-aa60-7d992d73c25b", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:46Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:46Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "fa600fa5-0a94-414c-aa60-7d992d73c25b", "secret": "model-catalog-postgres"} 2026-09-11T12:38:46Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "fa600fa5-0a94-414c-aa60-7d992d73c25b"} 2026-09-11T12:38:46Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "197911ed-66a1-403b-b7c3-573f7e93201d", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:46Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:46Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "197911ed-66a1-403b-b7c3-573f7e93201d", "secret": "model-catalog-postgres"} 2026-09-11T12:38:46Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "197911ed-66a1-403b-b7c3-573f7e93201d"} 2026-09-11T12:38:46Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "4b13c4a8-93ed-432e-b715-424be7701bfd", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:46Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:46Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "4b13c4a8-93ed-432e-b715-424be7701bfd", "secret": "model-catalog-postgres"} 2026-09-11T12:38:46Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "4b13c4a8-93ed-432e-b715-424be7701bfd"} 2026-09-11T12:38:46Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "8f1c8e11-2798-47dc-ad97-d436a3a354ea", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:46Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:46Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "8f1c8e11-2798-47dc-ad97-d436a3a354ea", "secret": "model-catalog-postgres"} 2026-09-11T12:38:46Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "8f1c8e11-2798-47dc-ad97-d436a3a354ea"} 2026-09-11T12:38:47Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "6acfafdf-38ea-47d6-a54c-21fbd563a1cb", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:47Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:47Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "6acfafdf-38ea-47d6-a54c-21fbd563a1cb", "secret": "model-catalog-postgres"} 2026-09-11T12:38:47Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "6acfafdf-38ea-47d6-a54c-21fbd563a1cb"} 2026-09-11T12:38:47Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "ccf34496-ba1b-43ca-866c-87e6531765d7", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:47Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:47Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "ccf34496-ba1b-43ca-866c-87e6531765d7", "secret": "model-catalog-postgres"} 2026-09-11T12:38:47Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "ccf34496-ba1b-43ca-866c-87e6531765d7"} 2026-09-11T12:38:49Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "d3030603-8ebe-4c71-ad8b-5b4d43f09369", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:49Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:49Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "d3030603-8ebe-4c71-ad8b-5b4d43f09369", "secret": "model-catalog-postgres"} 2026-09-11T12:38:49Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "d3030603-8ebe-4c71-ad8b-5b4d43f09369"} 2026-09-11T12:38:51Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "055a4ff7-e9c4-4a54-9ccf-a6f7b4edd389", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:51Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:51Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "055a4ff7-e9c4-4a54-9ccf-a6f7b4edd389", "secret": "model-catalog-postgres"} 2026-09-11T12:38:51Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "055a4ff7-e9c4-4a54-9ccf-a6f7b4edd389"} 2026-09-11T12:38:57Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "143d93ee-7b7a-4d84-a21c-03e9c4432795", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:57Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:57Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "143d93ee-7b7a-4d84-a21c-03e9c4432795", "secret": "model-catalog-postgres"} 2026-09-11T12:38:57Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "143d93ee-7b7a-4d84-a21c-03e9c4432795"} 2026-09-11T12:38:57Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "08bdfb19-0a5d-4cff-acce-714caf25c826", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:38:57Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:38:57Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "08bdfb19-0a5d-4cff-acce-714caf25c826", "secret": "model-catalog-postgres"} 2026-09-11T12:38:57Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "08bdfb19-0a5d-4cff-acce-714caf25c826"} 2026-09-11T12:39:10Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "4a939ec2-db5b-4811-af02-8b416aa7a0ad", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:39:10Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:39:10Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "4a939ec2-db5b-4811-af02-8b416aa7a0ad", "secret": "model-catalog-postgres"} 2026-09-11T12:39:10Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "4a939ec2-db5b-4811-af02-8b416aa7a0ad"} 2026-09-11T12:39:19Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "7de0eef8-ecdd-4cb3-ada9-ec1d894d9317", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:39:19Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:39:19Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "7de0eef8-ecdd-4cb3-ada9-ec1d894d9317", "secret": "model-catalog-postgres"} 2026-09-11T12:39:19Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "7de0eef8-ecdd-4cb3-ada9-ec1d894d9317"} 2026-09-11T12:39:19Z INFO controller-runtime.cache Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T12:39:19Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "da1fb776-c41c-4677-a85b-4648ad2d27d2", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:39:19Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:39:19Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "da1fb776-c41c-4677-a85b-4648ad2d27d2", "secret": "model-catalog-postgres"} 2026-09-11T12:39:19Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "da1fb776-c41c-4677-a85b-4648ad2d27d2"} 2026-09-11T12:39:20Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "cfc7c09c-f440-4f0b-9864-016d13a15ee6", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:39:20Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:39:20Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "cfc7c09c-f440-4f0b-9864-016d13a15ee6", "secret": "model-catalog-postgres"} 2026-09-11T12:39:20Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "cfc7c09c-f440-4f0b-9864-016d13a15ee6"} 2026-09-11T12:39:21Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "49ed037d-fb3d-420e-a15b-e3c1a5da6751", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:39:21Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:39:21Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "49ed037d-fb3d-420e-a15b-e3c1a5da6751", "secret": "model-catalog-postgres"} 2026-09-11T12:39:22Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "49ed037d-fb3d-420e-a15b-e3c1a5da6751"} 2026-09-11T12:39:23Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "56888d83-48da-4f94-8901-477ce4587a85", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:39:23Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:39:23Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "56888d83-48da-4f94-8901-477ce4587a85", "secret": "model-catalog-postgres"} 2026-09-11T12:39:23Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "56888d83-48da-4f94-8901-477ce4587a85"} 2026-09-11T12:39:24Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "25e840a6-8fe4-4d4b-9f76-ede68d629f4c", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:39:24Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:39:24Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "25e840a6-8fe4-4d4b-9f76-ede68d629f4c", "secret": "model-catalog-postgres"} 2026-09-11T12:39:24Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "25e840a6-8fe4-4d4b-9f76-ede68d629f4c"} 2026-09-11T12:39:24Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f2d05819-58ba-46c7-a0bb-2be027a01d14", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T12:39:24Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T12:39:24Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f2d05819-58ba-46c7-a0bb-2be027a01d14", "secret": "model-catalog-postgres"} 2026-09-11T12:39:24Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "f2d05819-58ba-46c7-a0bb-2be027a01d14"} 2026-09-11T12:47:23Z INFO controller-runtime.cache Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T12:54:05Z INFO controller-runtime.cache Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T12:59:13Z INFO controller-runtime.cache Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T13:05:01Z INFO controller-runtime.cache Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T13:10:27Z INFO controller-runtime.cache Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T13:13:11Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "faf97859-41bb-4a18-9d88-139af579b57d", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:13:11Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:13:11Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "faf97859-41bb-4a18-9d88-139af579b57d", "secret": "model-catalog-postgres"} 2026-09-11T13:13:11Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "faf97859-41bb-4a18-9d88-139af579b57d"} 2026-09-11T13:13:11Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "28ca0619-9dc1-468c-8c1f-e87882f6fc77", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:13:11Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:13:11Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "28ca0619-9dc1-468c-8c1f-e87882f6fc77", "secret": "model-catalog-postgres"} 2026-09-11T13:13:11Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "28ca0619-9dc1-468c-8c1f-e87882f6fc77"} 2026-09-11T13:13:39Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "d7ff9ea9-9ccb-44cb-bfd3-b5eba09f9bd6", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:13:39Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:13:39Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "d7ff9ea9-9ccb-44cb-bfd3-b5eba09f9bd6", "secret": "model-catalog-postgres"} 2026-09-11T13:13:39Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "d7ff9ea9-9ccb-44cb-bfd3-b5eba09f9bd6"} 2026-09-11T13:13:44Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "70aeb6bd-100d-4882-91e2-95ad5fd50748", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:13:44Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:13:44Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "70aeb6bd-100d-4882-91e2-95ad5fd50748", "secret": "model-catalog-postgres"} 2026-09-11T13:13:44Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "70aeb6bd-100d-4882-91e2-95ad5fd50748"} 2026-09-11T13:13:44Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "eb5ffe72-d4b7-493b-8980-19d71b551c11", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:13:44Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:13:44Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "eb5ffe72-d4b7-493b-8980-19d71b551c11", "secret": "model-catalog-postgres"} 2026-09-11T13:13:44Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "eb5ffe72-d4b7-493b-8980-19d71b551c11"} 2026-09-11T13:14:13Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "49872314-b1db-4311-9ab2-9cfa66896838", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:14:13Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:14:13Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "49872314-b1db-4311-9ab2-9cfa66896838", "secret": "model-catalog-postgres"} 2026-09-11T13:14:13Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "49872314-b1db-4311-9ab2-9cfa66896838"} 2026-09-11T13:14:16Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "c5eeb261-0c35-412d-80f3-1de0a88ba7d4", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:14:16Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:14:16Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "c5eeb261-0c35-412d-80f3-1de0a88ba7d4", "secret": "model-catalog-postgres"} 2026-09-11T13:14:16Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "c5eeb261-0c35-412d-80f3-1de0a88ba7d4"} 2026-09-11T13:14:16Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "a9fd016b-647e-4d86-81d0-f4a01d9d5da5", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:14:16Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:14:16Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "a9fd016b-647e-4d86-81d0-f4a01d9d5da5", "secret": "model-catalog-postgres"} 2026-09-11T13:14:16Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "a9fd016b-647e-4d86-81d0-f4a01d9d5da5"} 2026-09-11T13:15:17Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "94792a3d-b509-45cf-aa71-538a59dfb1cb", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:15:17Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:15:17Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "94792a3d-b509-45cf-aa71-538a59dfb1cb", "secret": "model-catalog-postgres"} 2026-09-11T13:15:17Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "94792a3d-b509-45cf-aa71-538a59dfb1cb"} 2026-09-11T13:16:16Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "15dfa9fb-313f-4bbc-abf0-ca7e92b5a71e", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:16:16Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:16:16Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "15dfa9fb-313f-4bbc-abf0-ca7e92b5a71e", "secret": "model-catalog-postgres"} 2026-09-11T13:16:16Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "15dfa9fb-313f-4bbc-abf0-ca7e92b5a71e"} 2026-09-11T13:16:16Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "05e060b6-7595-43f0-943b-54a0089d4c33", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:16:16Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:16:16Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "05e060b6-7595-43f0-943b-54a0089d4c33", "secret": "model-catalog-postgres"} 2026-09-11T13:16:16Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "05e060b6-7595-43f0-943b-54a0089d4c33"} 2026-09-11T13:16:40Z INFO controller-runtime.cache Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-11T13:17:12Z INFO Reconciling catalog {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "be6c6122-e42b-4b94-9218-7da3009b30ec", "name": "catalog", "namespace": "rhoai-model-registries"} 2026-09-11T13:17:12Z INFO catalog-controller Found admin groups from auth CR {"groups": ["rhods-admins"]} 2026-09-11T13:17:12Z DEBUG Postgres secret already exists, reconciling {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "be6c6122-e42b-4b94-9218-7da3009b30ec", "secret": "model-catalog-postgres"} 2026-09-11T13:17:12Z INFO Creating or updating postgres NetworkPolicy {"controller": "catalog", "controllerGroup": "aihub.opendatahub.io", "controllerKind": "Catalog", "Catalog": {"name":"catalog","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "catalog", "reconcileID": "be6c6122-e42b-4b94-9218-7da3009b30ec"} ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-operator-6b54b88779-zqz88.log ==================== 2026-09-11T12:37:53Z INFO setup Detected platform {"platform": "OpenShift AI Self-Managed"} 2026-09-11T12:37:53Z INFO setup Starting manager 2026-09-11T12:37:53Z INFO controller-runtime.metrics Starting metrics server 2026-09-11T12:37:53Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} I0911 12:37:53.955227 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="redhat-ods-applications/dashboard.components.platform.opendatahub.io" I0911 12:37:53.985061 1 leaderelection.go:272] "Successfully acquired lease" lock="redhat-ods-applications/dashboard.components.platform.opendatahub.io" 2026-09-11T12:37:53Z DEBUG events dashboard-operator-6b54b88779-zqz88_826edea5-9f3a-499b-a97f-c908bdc59760 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"redhat-ods-applications","name":"dashboard.components.platform.opendatahub.io","uid":"6813a8cb-25bb-4a51-879a-7b4cb71540b7","apiVersion":"coordination.k8s.io/v1","resourceVersion":"28167"}, "reason": "LeaderElection"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "source": "kind source: *v1.PodDisruptionBudget"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "source": "kind source: *v1alpha1.Dashboard"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "source": "kind source: *v1.Deployment"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "source": "kind source: *v1.Service"} 2026-09-11T12:37:53Z INFO Starting EventSource {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "source": "kind source: *v1.ConfigMap"} 2026-09-11T12:37:54Z INFO Starting Controller {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard"} 2026-09-11T12:37:54Z INFO Starting workers {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "worker count": 1} 2026-09-11T12:37:54Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "09d0b9ce-2ad8-4d70-8a6c-949715ff5d73", "name": "default-dashboard"} 2026-09-11T12:37:54Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "bd320dc7-8950-473f-aab6-ffa290a10110", "name": "default-dashboard"} 2026-09-11T12:37:54Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "bd320dc7-8950-473f-aab6-ffa290a10110"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:37:54Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8443", "secure": true} 2026-09-11T12:37:55Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "bd320dc7-8950-473f-aab6-ffa290a10110", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:55Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "890882c1-3c3b-4e97-8a30-853fc9218787", "name": "default-dashboard"} 2026-09-11T12:37:55Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "890882c1-3c3b-4e97-8a30-853fc9218787"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:37:55Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "890882c1-3c3b-4e97-8a30-853fc9218787", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:55Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "3a7a3cff-e1c8-4940-b763-ad387f3465a7", "name": "default-dashboard"} 2026-09-11T12:37:56Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "3a7a3cff-e1c8-4940-b763-ad387f3465a7"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:37:56Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "3a7a3cff-e1c8-4940-b763-ad387f3465a7", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:56Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "ef93449c-7283-45e4-9698-7a8978d61518", "name": "default-dashboard"} 2026-09-11T12:37:56Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "ef93449c-7283-45e4-9698-7a8978d61518"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:37:57Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "ef93449c-7283-45e4-9698-7a8978d61518", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:57Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0033b436-73cd-4213-8475-57d41d2ece5d", "name": "default-dashboard"} 2026-09-11T12:37:57Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0033b436-73cd-4213-8475-57d41d2ece5d"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:37:57Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0033b436-73cd-4213-8475-57d41d2ece5d", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:57Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4817a847-8334-4971-9696-0962e94490b6", "name": "default-dashboard"} 2026-09-11T12:37:57Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4817a847-8334-4971-9696-0962e94490b6"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:37:58Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4817a847-8334-4971-9696-0962e94490b6", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:58Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c18cd8a8-607e-4c6e-beb0-2d13fbd7806d", "name": "default-dashboard"} 2026-09-11T12:37:58Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c18cd8a8-607e-4c6e-beb0-2d13fbd7806d"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:37:58Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c18cd8a8-607e-4c6e-beb0-2d13fbd7806d", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:37:59Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "78591d10-5c61-485c-a7a6-2e5333341018", "name": "default-dashboard"} 2026-09-11T12:37:59Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "78591d10-5c61-485c-a7a6-2e5333341018"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:37:59Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "78591d10-5c61-485c-a7a6-2e5333341018", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:00Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "5f2caea3-a87e-4d1d-abd5-a16d4f82ce39", "name": "default-dashboard"} 2026-09-11T12:38:00Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "5f2caea3-a87e-4d1d-abd5-a16d4f82ce39"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:00Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "5f2caea3-a87e-4d1d-abd5-a16d4f82ce39", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:02Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d0ee37b5-c847-478f-b099-e6a71f0481c6", "name": "default-dashboard"} 2026-09-11T12:38:02Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d0ee37b5-c847-478f-b099-e6a71f0481c6"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:02Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d0ee37b5-c847-478f-b099-e6a71f0481c6", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:05Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "ce6c35bd-f1f5-4f9e-b7bd-2264ccc1f803", "name": "default-dashboard"} 2026-09-11T12:38:05Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "ce6c35bd-f1f5-4f9e-b7bd-2264ccc1f803"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:05Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "ce6c35bd-f1f5-4f9e-b7bd-2264ccc1f803", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:10Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0b700c93-70b9-400d-b30f-730668b57c9f", "name": "default-dashboard"} 2026-09-11T12:38:10Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0b700c93-70b9-400d-b30f-730668b57c9f"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:11Z ERROR Reconciler error {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0b700c93-70b9-400d-b30f-730668b57c9f", "error": "failed to deploy core resources: failure deploying resource redhat-ods-applications/rhods-dashboard: apply failed /v1, Kind=ServiceAccount: unable to patch /v1, Kind=ServiceAccount redhat-ods-applications/rhods-dashboard: serviceaccounts \"rhods-dashboard\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion components.platform.opendatahub.io/v1alpha1 Kind Dashboard: no matches for kind \"Dashboard\" in version \"components.platform.opendatahub.io/v1alpha1\""} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:21Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "name": "default-dashboard"} 2026-09-11T12:38:21Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:27Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:38:28Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:38:28Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:38:28Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "module": "automl", "slug": "automl"} 2026-09-11T12:38:29Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:38:29Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "module": "maas", "slug": "maas"} 2026-09-11T12:38:29Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "module": "autorag", "slug": "autorag"} 2026-09-11T12:38:30Z INFO Patched federation config hash on deployment {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "deployment": "rhods-dashboard", "hash": "adfc49524183723cc425a29fd0054b31335eeccc681f21cf26ddda40ce727d68"} 2026-09-11T12:38:30Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "01e6e7b0-8779-4ffa-87e3-1efafb4c8a04", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:38:30Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "name": "default-dashboard"} 2026-09-11T12:38:30Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:34Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:38:34Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "module": "maas", "slug": "maas"} 2026-09-11T12:38:34Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "module": "autorag", "slug": "autorag"} 2026-09-11T12:38:35Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:38:35Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:38:35Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:38:35Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "module": "automl", "slug": "automl"} 2026-09-11T12:38:35Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:38:35Z ERROR Failed to update status {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a54a1971-cebe-4f2e-8e3d-efa475ed80c4", "error": "Operation cannot be fulfilled on dashboards.components.platform.opendatahub.io \"default-dashboard\": the object has been modified; please apply your changes to the latest version and try again"} github.com/opendatahub-io/odh-dashboard/dashboard-operator/internal/controller.(*DashboardReconciler).Reconcile /usr/src/app/internal/controller/dashboard_reconciler.go:225 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:38:35Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c", "name": "default-dashboard"} 2026-09-11T12:38:36Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:40Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c", "module": "autorag", "slug": "autorag"} 2026-09-11T12:38:40Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:38:40Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:38:40Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:38:41Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c", "module": "automl", "slug": "automl"} 2026-09-11T12:38:41Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:38:41Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c", "module": "maas", "slug": "maas"} 2026-09-11T12:38:41Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "22bbd994-7d29-40b1-ab45-485b371b631c", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:38:41Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26", "name": "default-dashboard"} 2026-09-11T12:38:41Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:45Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:38:45Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26", "module": "maas", "slug": "maas"} 2026-09-11T12:38:45Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26", "module": "autorag", "slug": "autorag"} 2026-09-11T12:38:46Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:38:46Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:38:46Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:38:46Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26", "module": "automl", "slug": "automl"} 2026-09-11T12:38:46Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "0a9b3e26-220f-4536-8271-9ea124f56b26", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:38:46Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8", "name": "default-dashboard"} 2026-09-11T12:38:46Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:50Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:38:50Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8", "module": "automl", "slug": "automl"} 2026-09-11T12:38:51Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:38:51Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8", "module": "maas", "slug": "maas"} 2026-09-11T12:38:51Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8", "module": "autorag", "slug": "autorag"} 2026-09-11T12:38:51Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:38:51Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:38:51Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "d3fa271e-9164-460c-8f06-498b28f6e1d8", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:38:52Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6", "name": "default-dashboard"} 2026-09-11T12:38:52Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:38:55Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6", "module": "autorag", "slug": "autorag"} 2026-09-11T12:38:55Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:38:55Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:38:56Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:38:56Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6", "module": "automl", "slug": "automl"} 2026-09-11T12:38:56Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:38:56Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6", "module": "maas", "slug": "maas"} 2026-09-11T12:38:56Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6a97aa9f-2780-4c8b-9c86-cb6478b25fc6", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:38:56Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577", "name": "default-dashboard"} 2026-09-11T12:38:56Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:39:00Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577", "module": "autorag", "slug": "autorag"} 2026-09-11T12:39:00Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:39:00Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:39:00Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:39:01Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577", "module": "automl", "slug": "automl"} 2026-09-11T12:39:01Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:39:01Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577", "module": "maas", "slug": "maas"} 2026-09-11T12:39:01Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "6830f5bb-7b5e-47af-beca-c52406798577", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:39:01Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37", "name": "default-dashboard"} 2026-09-11T12:39:01Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:39:05Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37", "module": "automl", "slug": "automl"} 2026-09-11T12:39:05Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:39:05Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37", "module": "maas", "slug": "maas"} 2026-09-11T12:39:05Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37", "module": "autorag", "slug": "autorag"} 2026-09-11T12:39:06Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:39:06Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:39:06Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:39:06Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2ade49a2-fa04-4be9-96f2-e18bb12aea37", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:39:06Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde", "name": "default-dashboard"} 2026-09-11T12:39:06Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:39:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:39:10Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:39:10Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:39:10Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde", "module": "automl", "slug": "automl"} 2026-09-11T12:39:10Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:39:11Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde", "module": "maas", "slug": "maas"} 2026-09-11T12:39:11Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde", "module": "autorag", "slug": "autorag"} 2026-09-11T12:39:11Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "86aee0d8-a50c-4b9b-a29f-a4bff1882dde", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:39:11Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "name": "default-dashboard"} 2026-09-11T12:39:11Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:39:14Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:39:15Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "module": "maas", "slug": "maas"} 2026-09-11T12:39:15Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "module": "autorag", "slug": "autorag"} 2026-09-11T12:39:15Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:39:15Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:39:15Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:39:16Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "module": "automl", "slug": "automl"} 2026-09-11T12:39:16Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:39:16Z ERROR Failed to update status {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "2d970c22-8eb0-4914-a6bc-eceb9b6bffa1", "error": "Operation cannot be fulfilled on dashboards.components.platform.opendatahub.io \"default-dashboard\": the object has been modified; please apply your changes to the latest version and try again"} github.com/opendatahub-io/odh-dashboard/dashboard-operator/internal/controller.(*DashboardReconciler).Reconcile /usr/src/app/internal/controller/dashboard_reconciler.go:225 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:39:16Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa", "name": "default-dashboard"} 2026-09-11T12:39:16Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:39:19Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:39:20Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:39:20Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:39:20Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa", "module": "automl", "slug": "automl"} 2026-09-11T12:39:20Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:39:20Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa", "module": "maas", "slug": "maas"} 2026-09-11T12:39:21Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa", "module": "autorag", "slug": "autorag"} 2026-09-11T12:39:21Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "e42e8c44-361f-4316-99ba-579164f35dfa", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:39:21Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "name": "default-dashboard"} 2026-09-11T12:39:21Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:39:24Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:39:24Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "module": "maas", "slug": "maas"} 2026-09-11T12:39:25Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "module": "autorag", "slug": "autorag"} 2026-09-11T12:39:25Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:39:25Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:39:25Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:39:26Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "module": "automl", "slug": "automl"} 2026-09-11T12:39:26Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:39:26Z ERROR Failed to update status {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "23b4ac86-bd92-42cd-bbcf-f37d997b6fcc", "error": "Operation cannot be fulfilled on dashboards.components.platform.opendatahub.io \"default-dashboard\": the object has been modified; please apply your changes to the latest version and try again"} github.com/opendatahub-io/odh-dashboard/dashboard-operator/internal/controller.(*DashboardReconciler).Reconcile /usr/src/app/internal/controller/dashboard_reconciler.go:225 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312 2026-09-11T12:39:26Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3", "name": "default-dashboard"} 2026-09-11T12:39:26Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:39:29Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:39:29Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3", "module": "maas", "slug": "maas"} 2026-09-11T12:39:30Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3", "module": "autorag", "slug": "autorag"} 2026-09-11T12:39:30Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:39:30Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:39:30Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:39:30Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3", "module": "automl", "slug": "automl"} 2026-09-11T12:39:31Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "c232fde8-8972-4307-8b33-f8ddbab4dfb3", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:39:51Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71", "name": "default-dashboard"} 2026-09-11T12:39:51Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:39:55Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:39:55Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71", "module": "maas", "slug": "maas"} 2026-09-11T12:39:55Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71", "module": "autorag", "slug": "autorag"} 2026-09-11T12:39:56Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:39:56Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:39:56Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:39:56Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71", "module": "automl", "slug": "automl"} 2026-09-11T12:39:56Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "a816b286-8a8e-4091-896e-da6d4a8a8c71", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:39:59Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1", "name": "default-dashboard"} 2026-09-11T12:39:59Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:40:03Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:40:03Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1", "module": "maas", "slug": "maas"} 2026-09-11T12:40:03Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1", "module": "autorag", "slug": "autorag"} 2026-09-11T12:40:04Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:40:04Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:40:04Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:40:04Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1", "module": "automl", "slug": "automl"} 2026-09-11T12:40:04Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "566e901b-1aa4-4491-90be-b4547621b2f1", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:40:04Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb", "name": "default-dashboard"} 2026-09-11T12:40:04Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:40:08Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:40:08Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb", "module": "maas", "slug": "maas"} 2026-09-11T12:40:08Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb", "module": "autorag", "slug": "autorag"} 2026-09-11T12:40:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:40:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:40:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:40:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb", "module": "automl", "slug": "automl"} 2026-09-11T12:40:09Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "9502ef5c-3848-4b35-ab6b-b3b8b5c0e1cb", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:40:31Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79", "name": "default-dashboard"} 2026-09-11T12:40:31Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:40:34Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79", "module": "automl", "slug": "automl"} 2026-09-11T12:40:34Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:40:35Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79", "module": "maas", "slug": "maas"} 2026-09-11T12:40:35Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79", "module": "autorag", "slug": "autorag"} 2026-09-11T12:40:35Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:40:35Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:40:36Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:40:36Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7aed4785-7918-464b-bdcf-131b46112f79", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:45:36Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4", "name": "default-dashboard"} 2026-09-11T12:45:36Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:45:39Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:45:39Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4", "module": "maas", "slug": "maas"} 2026-09-11T12:45:40Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4", "module": "autorag", "slug": "autorag"} 2026-09-11T12:45:40Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:45:40Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:45:40Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:45:40Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4", "module": "automl", "slug": "automl"} 2026-09-11T12:45:40Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "b30ad3ea-0602-4b96-b7dc-0ca1197c6aa4", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:50:40Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d", "name": "default-dashboard"} 2026-09-11T12:50:41Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:50:44Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d", "module": "autorag", "slug": "autorag"} 2026-09-11T12:50:44Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:50:44Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:50:45Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:50:45Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d", "module": "automl", "slug": "automl"} 2026-09-11T12:50:45Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:50:45Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d", "module": "maas", "slug": "maas"} 2026-09-11T12:50:45Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "4a942fbd-0e67-485a-9226-cd4755bc990d", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T12:55:45Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a", "name": "default-dashboard"} 2026-09-11T12:55:45Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T12:55:49Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T12:55:49Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a", "module": "automl", "slug": "automl"} 2026-09-11T12:55:49Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a", "module": "genAi", "slug": "gen-ai"} 2026-09-11T12:55:49Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a", "module": "maas", "slug": "maas"} 2026-09-11T12:55:50Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a", "module": "autorag", "slug": "autorag"} 2026-09-11T12:55:50Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T12:55:50Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a", "module": "notebooks", "slug": "notebooks"} 2026-09-11T12:55:50Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "418236e5-243c-450a-b8e7-f40280d2cb2a", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T13:00:50Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70", "name": "default-dashboard"} 2026-09-11T13:00:50Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T13:00:54Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T13:00:54Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70", "module": "notebooks", "slug": "notebooks"} 2026-09-11T13:00:54Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T13:00:54Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70", "module": "automl", "slug": "automl"} 2026-09-11T13:00:55Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70", "module": "genAi", "slug": "gen-ai"} 2026-09-11T13:00:55Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70", "module": "maas", "slug": "maas"} 2026-09-11T13:00:55Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70", "module": "autorag", "slug": "autorag"} 2026-09-11T13:00:55Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "7cc8dc94-ef5b-46f4-adea-10853bf1af70", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T13:05:55Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804", "name": "default-dashboard"} 2026-09-11T13:05:55Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T13:05:59Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804", "module": "notebooks", "slug": "notebooks"} 2026-09-11T13:05:59Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T13:05:59Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804", "module": "automl", "slug": "automl"} 2026-09-11T13:05:59Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804", "module": "genAi", "slug": "gen-ai"} 2026-09-11T13:05:59Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804", "module": "maas", "slug": "maas"} 2026-09-11T13:06:00Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804", "module": "autorag", "slug": "autorag"} 2026-09-11T13:06:00Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T13:06:00Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "cede9200-c63d-4e2c-be3b-34b455dff804", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T13:11:00Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2", "name": "default-dashboard"} 2026-09-11T13:11:00Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T13:11:03Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T13:11:04Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2", "module": "notebooks", "slug": "notebooks"} 2026-09-11T13:11:04Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T13:11:04Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2", "module": "automl", "slug": "automl"} 2026-09-11T13:11:04Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2", "module": "genAi", "slug": "gen-ai"} 2026-09-11T13:11:05Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2", "module": "maas", "slug": "maas"} 2026-09-11T13:11:05Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2", "module": "autorag", "slug": "autorag"} 2026-09-11T13:11:05Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "eb71fe92-63db-4914-9a02-e7c12f5763d2", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} 2026-09-11T13:16:05Z INFO Reconciling Dashboard {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05", "name": "default-dashboard"} 2026-09-11T13:16:05Z INFO Cleaned up sidecar-specific resources {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. 2026-09-11T13:16:08Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05", "module": "agentOps", "slug": "agent-ops"} 2026-09-11T13:16:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05", "module": "notebooks", "slug": "notebooks"} 2026-09-11T13:16:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05", "module": "modelRegistry", "slug": "model-registry"} 2026-09-11T13:16:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05", "module": "automl", "slug": "automl"} 2026-09-11T13:16:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05", "module": "genAi", "slug": "gen-ai"} 2026-09-11T13:16:09Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05", "module": "maas", "slug": "maas"} 2026-09-11T13:16:10Z INFO Deployed module {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05", "module": "autorag", "slug": "autorag"} 2026-09-11T13:16:10Z INFO Standalone mode reconciled successfully {"controller": "dashboard", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Dashboard", "Dashboard": {"name":"default-dashboard"}, "namespace": "", "name": "default-dashboard", "reconcileID": "91d65bde-91b7-4881-b6af-0411d0806d05", "url": "https://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/", "modules": 9} ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-56cff8d89b-8dctm.log ==================== 2026/09/11 12:37:56 [notice] 1#1: using the "epoll" event method 2026/09/11 12:37:56 [notice] 1#1: nginx/1.26.3 2026/09/11 12:37:56 [notice] 1#1: built by gcc 11.5.0 20240719 (Red Hat 11.5.0-14) (GCC) 2026/09/11 12:37:56 [notice] 1#1: OS: Linux 5.14.0-570.135.1.el9_6.x86_64 2026/09/11 12:37:56 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1024:524288 2026/09/11 12:37:56 [notice] 1#1: start worker processes 2026/09/11 12:37:56 [notice] 1#1: start worker process 12 2026/09/11 12:37:56 [notice] 1#1: start worker process 13 ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-76f9cb9bsqnf.log ==================== 2026-09-11T12:37:47Z INFO setup MLflow reads use direct API access (bypass controller-runtime client cache) 2026-09-11T12:37:47Z INFO setup starting manager 2026-09-11T12:37:47Z INFO controller-runtime.metrics Starting metrics server 2026-09-11T12:37:47Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} 2026-09-11T12:37:47Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "tlssecurityprofilewatcher", "source": "kind source: *v1.APIServer"} I0911 12:37:47.816636 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="redhat-ods-applications/f9eb95d5.opendatahub.io" I0911 12:37:47.850523 1 leaderelection.go:272] "Successfully acquired lease" lock="redhat-ods-applications/f9eb95d5.opendatahub.io" 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Secret"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Deployment"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Secret"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.ConfigMap"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.DataSciencePipelinesApplication"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Service"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.ServiceAccount"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.PersistentVolumeClaim"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.NetworkPolicy"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Pod"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Role"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.RoleBinding"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Route"} 2026-09-11T12:37:47Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.ConfigMap"} 2026-09-11T12:37:48Z INFO Starting Controller {"controller": "tlssecurityprofilewatcher"} 2026-09-11T12:37:48Z INFO Starting workers {"controller": "tlssecurityprofilewatcher", "worker count": 1} 2026-09-11T12:37:48Z INFO Starting Controller {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication"} 2026-09-11T12:37:48Z INFO Starting workers {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "worker count": 10} ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-5fc6bb8d67-fngvd.log ==================== 2026-09-11T12:38:18Z INFO setup FeatureStore installation metrics enabled 2026-09-11T12:38:18Z INFO setup Notebook CRD validated successfully {"CRD": "notebooks.kubeflow.org", "GVK": "kubeflow.org/v1, Kind=Notebook"} 2026-09-11T12:38:18Z INFO setup Notebook ConfigMap controller setup completed successfully 2026-09-11T12:38:18Z INFO setup starting manager 2026-09-11T12:38:18Z INFO controller-runtime.metrics Starting metrics server 2026-09-11T12:38:18Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "tlssecurityprofilewatcher", "source": "kind source: *v1.APIServer"} I0911 12:38:18.848162 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="redhat-ods-applications/66483dbd.feast.dev" I0911 12:38:18.877255 1 leaderelection.go:272] "Successfully acquired lease" lock="redhat-ods-applications/66483dbd.feast.dev" 2026-09-11T12:38:18Z DEBUG events feast-operator-controller-manager-5fc6bb8d67-fngvd_9ccba4ea-ccf7-4d53-ad8e-dc186877259f became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"redhat-ods-applications","name":"66483dbd.feast.dev","uid":"1f218a87-70cc-4ab5-80a1-8f08f1c1f216","apiVersion":"coordination.k8s.io/v1","resourceVersion":"28865"}, "reason": "LeaderElection"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.FeatureStore"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "source": "kind source: *v1.FeatureStore"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.FeatureStore"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.ConfigMap"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.Deployment"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.Service"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.PersistentVolumeClaim"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.ServiceAccount"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.RoleBinding"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.Role"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.CronJob"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v2.HorizontalPodAutoscaler"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.PodDisruptionBudget"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *v1.Route"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "source": "kind source: *unstructured.Unstructured"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "source": "kind source: *unstructured.Unstructured"} 2026-09-11T12:38:18Z INFO Starting EventSource {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "source": "kind source: *v1.ConfigMap"} 2026-09-11T12:38:18Z INFO Starting Controller {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook"} 2026-09-11T12:38:18Z INFO Starting workers {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "worker count": 1} 2026-09-11T12:38:19Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8443", "secure": true} 2026-09-11T12:38:19Z INFO Starting Controller {"controller": "tlssecurityprofilewatcher"} 2026-09-11T12:38:19Z INFO Starting workers {"controller": "tlssecurityprofilewatcher", "worker count": 1} 2026-09-11T12:38:19Z DEBUG Reconciling APIServer TLS profile {"controller": "tlssecurityprofilewatcher", "reconcileID": "a595e16c-6b3e-4dd7-a704-07f07cf8054d", "name": "cluster"} 2026-09-11T12:38:19Z DEBUG Finished reconciling APIServer TLS profile {"controller": "tlssecurityprofilewatcher", "reconcileID": "a595e16c-6b3e-4dd7-a704-07f07cf8054d", "name": "cluster"} 2026-09-11T12:38:19Z INFO Starting Controller {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore"} 2026-09-11T12:38:19Z INFO Starting workers {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "worker count": 1} 2026-09-11T13:09:55Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "143e8907-d409-4896-8b4a-8a78c52f6ed2", "label": "opendatahub.io/feast-integration", "value": ""} 2026-09-11T13:09:55Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "1dc403b1-648d-46e4-b495-7bc0cefdef45", "label": "opendatahub.io/feast-integration", "value": ""} 2026-09-11T13:09:56Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "5fe7eb1e-0fdc-4143-aca6-1fdc6905c1e4", "label": "opendatahub.io/feast-integration", "value": ""} 2026-09-11T13:09:56Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "640dedd3-8f2c-4c30-bc18-bd4da882da54", "label": "opendatahub.io/feast-integration", "value": ""} 2026-09-11T13:10:37Z DEBUG Notebook is being deleted, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "12b7dd79-0da9-4be8-9dd3-d74ef269d1fb"} 2026-09-11T13:10:37Z DEBUG Notebook is being deleted, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "48291e70-2002-4194-9b12-6404f8176ccd"} 2026-09-11T13:10:37Z DEBUG Notebook not found, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "a74e4b3a-7e27-4271-b6d9-b805725dbe01"} ==================== WORKLOAD POD LOG: redhat-ods-applications/gen-ai-ui-556b57bc96-9rrlc.log ==================== WARNING: proto: file "opentelemetry/proto/trace/v1/trace.proto" is already registered previously from: "github.com/opendatahub-io/mlflow-go/internal/gen/otelpb" currently from: "go.opentelemetry.io/proto/otlp/trace/v1" See https://protobuf.dev/reference/go/faq#namespace-conflict WARNING: proto: file "opentelemetry/proto/trace/v1/trace.proto" has a name conflict over opentelemetry.proto.trace.v1.Span previously from: "github.com/opendatahub-io/mlflow-go/internal/gen/otelpb" currently from: "go.opentelemetry.io/proto/otlp/trace/v1" See https://protobuf.dev/reference/go/faq#namespace-conflict WARNING: proto: message opentelemetry.proto.trace.v1.Span is already registered previously from: "github.com/opendatahub-io/mlflow-go/internal/gen/otelpb" currently from: "go.opentelemetry.io/proto/otlp/trace/v1" See https://protobuf.dev/reference/go/faq#namespace-conflict time=2026-09-11T12:38:42.329Z level=INFO msg="OTel tracing disabled (no collector endpoint configured or discovered)" time=2026-09-11T12:38:42.783Z level=INFO msg="Added CA bundle" path=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem time=2026-09-11T12:38:42.783Z level=INFO msg="Added CA bundle" path=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt time=2026-09-11T12:38:42.783Z level=INFO msg="Added CA bundle" path=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt time=2026-09-11T12:38:42.786Z level=INFO msg="Added CA bundle" path=/etc/pki/tls/certs/odh-trusted-ca-bundle.crt time=2026-09-11T12:38:42.786Z level=INFO msg="Detected dashboard namespace" namespace=redhat-ods-applications time=2026-09-11T12:38:42.786Z level=INFO msg="Using real LlamaStack client factory" url="" time=2026-09-11T12:38:42.786Z level=INFO msg="Using real NeMo Guardrails client factory" url="" time=2026-09-11T12:38:43.792Z level=INFO msg="platform OpenTelemetryCollector CR not found, trace route management disabled" error="failed to list OpenTelemetryCollector CRs for platform collector discovery: the server could not find the requested resource" time=2026-09-11T12:38:43.795Z level=INFO msg="SA client initialized for pgvector provisioning" time=2026-09-11T12:38:43.806Z level=WARN msg="MLflow URL not configured and auto-discovery failed, MLflow endpoints will return 503" time=2026-09-11T12:38:43.816Z level=WARN msg="Failed to discover MLflow external URL for code export" error="failed to list MLflow CRs: the server could not find the requested resource" time=2026-09-11T12:38:43.817Z level=INFO msg="Using real BFF client factory" maasServiceName=odh-dashboard-maas-ui maasServicePort=8243 maasDevURL="" mlflowServiceName=odh-dashboard-mlflow-ui mlflowServicePort=8343 mlflowDevURL="" time=2026-09-11T12:38:43.817Z level=INFO msg="Initialized file upload job tracker" time=2026-09-11T12:38:43.829Z level=INFO msg="Cached cluster domain for MaaS autodiscovery" domain=apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev time=2026-09-11T12:38:43.829Z level=INFO msg="starting server" addr=:8143 "TLS enabled"=true path_prefix=/gen-ai swagger_ui=/gen-ai/swagger-ui ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-controller-manager-58c6879fc6-n2gv5.log ==================== {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"Setting up client for manager"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"tls","msg":"Resolved TLS configuration from cluster security profile","adherence":"","type":"Intermediate","minVersion":771} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"Setting up manager"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"Registering Components."} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"Setting up controller schemes"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"Setting up v1beta1 controller"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"v1beta1Controllers.InferenceService","msg":"The InferenceService controller won't watch serving.knative.dev/v1/Service resources because the CRD is not available."} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"v1beta1Controllers.InferenceService","msg":"The InferenceService controller won't watch opentelemetry-collector resources because the CRD is not available."} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"Setting up v1beta1 TrainedModel controller"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"Setting up InferenceGraph controller"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"v1alpha1Controllers.InferenceGraph","msg":"The InferenceGraph controller won't watch serving.knative.dev/v1/Service resources because the CRD is not available."} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"setting up webhook server"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"registering webhooks to the webhook server"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-pods"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"registering cluster serving runtime validator webhook to the webhook server"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha1-clusterservingruntime"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"registering serving runtime validator webhook to the webhook server"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha1-servingruntime"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1alpha1, Kind=TrainedModel","path":"/validate-serving-kserve-io-v1alpha1-trainedmodel"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha1-trainedmodel"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1alpha1, Kind=InferenceGraph","path":"/validate-serving-kserve-io-v1alpha1-inferencegraph"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1alpha1-inferencegraph"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"serving.kserve.io/v1beta1, Kind=InferenceService","path":"/mutate-serving-kserve-io-v1beta1-inferenceservice"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-serving-kserve-io-v1beta1-inferenceservice"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"serving.kserve.io/v1beta1, Kind=InferenceService","path":"/validate-serving-kserve-io-v1beta1-inferenceservice"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-serving-kserve-io-v1beta1-inferenceservice"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"setup","msg":"Starting the Cmd."} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"tls-profile-watcher","controllerGroup":"config.openshift.io","controllerKind":"APIServer","source":"kind source: *v1.APIServer"} I0911 12:38:47.498344 1 leaderelection.go:257] attempting to acquire leader lease redhat-ods-applications/kserve-controller-manager-leader-lock... I0911 12:38:47.533345 1 leaderelection.go:271] successfully acquired lease redhat-ods-applications/kserve-controller-manager-leader-lock {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1alpha1.ClusterServingRuntime"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph","source":"kind source: *v1alpha1.InferenceGraph"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph","source":"kind source: *v1.Route"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1beta1.InferenceService"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1.Deployment"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1.Service"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1alpha1.ScaledObject"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1beta1.VirtualService"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1.Ingress"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1alpha1.ServingRuntime"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph","source":"kind source: *v1.Deployment"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting EventSource","controller":"trainedmodel","controllerGroup":"serving.kserve.io","controllerKind":"TrainedModel","source":"kind source: *v1alpha1.TrainedModel"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting Controller","controller":"tls-profile-watcher","controllerGroup":"config.openshift.io","controllerKind":"APIServer"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting workers","controller":"tls-profile-watcher","controllerGroup":"config.openshift.io","controllerKind":"APIServer","worker count":1} {"level":"info","ts":"2026-09-11T12:38:47Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting Controller","controller":"trainedmodel","controllerGroup":"serving.kserve.io","controllerKind":"TrainedModel"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting workers","controller":"trainedmodel","controllerGroup":"serving.kserve.io","controllerKind":"TrainedModel","worker count":1} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting Controller","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting workers","controller":"inferencegraph","controllerGroup":"serving.kserve.io","controllerKind":"InferenceGraph","worker count":1} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting Controller","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService"} {"level":"info","ts":"2026-09-11T12:38:47Z","msg":"Starting workers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","worker count":1} ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-module-controller-manager-6f7f44cc5-mnrt7.log ==================== {"level":"info","ts":"2026-09-11T12:37:58Z","logger":"setup","msg":"setting up kserve-module controller"} {"level":"info","ts":"2026-09-11T12:38:10Z","logger":"setup","msg":"starting manager"} {"level":"info","ts":"2026-09-11T12:38:10Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2026-09-11T12:38:10Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"starting server","name":"health probe","addr":"[::]:8081"} I0911 12:38:10.942923 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="redhat-ods-applications/kserve-module-controller-manager" I0911 12:38:10.975238 1 leaderelection.go:272] "Successfully acquired lease" lock="redhat-ods-applications/kserve-module-controller-manager" {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.Deployment"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1alpha1.Kserve"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.DaemonSet"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.ConfigMap"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.Secret"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.Service"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.NetworkPolicy"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.Role"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.RoleBinding"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.ClusterRole"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.ServiceAccount"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.PersistentVolume"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.PersistentVolumeClaim"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.ClusterRoleBinding"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.MutatingWebhookConfiguration"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.ValidatingWebhookConfiguration"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.ConfigMap"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-09-11T12:38:10Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Starting Controller","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"Starting workers","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","worker count":1} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"added module finalizer to Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6"} {"level":"info","ts":"2026-09-11T12:38:11Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","name":"default-kserve"} {"level":"info","ts":"2026-09-11T12:38:15Z","msg":"Skipping HWP migration: AcceleratorProfile CRD not found","group":"dashboard.opendatahub.io"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:38:38Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:38:38Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:38:38Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:38:39Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:38:40Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:38:40Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:38:40Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6"} {"level":"info","ts":"2026-09-11T12:38:40Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:38:40Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:38:40Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:38:40Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"console-dashboards","resources":4} {"level":"error","ts":"2026-09-11T12:38:45Z","msg":"component reconciliation failed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","component":"deploy","error":"applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"github.com/opendatahub-io/kserve-module/pkg/kservemodule.(*KserveModuleReconciler).Reconcile\n\t/go/src/github.com/opendatahub-io/kserve-module/pkg/kservemodule/reconciler.go:189\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-09-11T12:38:45Z","msg":"Reconciler error","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"0f4b78f3-4971-493d-ab6a-4b60240419b6","error":"reconciliation failed: deploy: applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-09-11T12:38:45Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","name":"default-kserve"} {"level":"info","ts":"2026-09-11T12:38:49Z","msg":"Starting EventSource","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2026-09-11T12:38:49Z","msg":"registered dynamic watch","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","gvk":"serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:38:50Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:38:51Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:38:51Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:38:51Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:38:51Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df"} {"level":"info","ts":"2026-09-11T12:38:51Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:38:51Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:38:51Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:38:51Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"console-dashboards","resources":4} {"level":"error","ts":"2026-09-11T12:38:54Z","msg":"component reconciliation failed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","component":"deploy","error":"applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"github.com/opendatahub-io/kserve-module/pkg/kservemodule.(*KserveModuleReconciler).Reconcile\n\t/go/src/github.com/opendatahub-io/kserve-module/pkg/kservemodule/reconciler.go:189\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-09-11T12:38:54Z","msg":"Reconciler error","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ab966934-56cb-4f9f-9e03-7f4b6548d0df","error":"reconciliation failed: deploy: applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-09-11T12:38:54Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","name":"default-kserve"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:38:58Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:38:59Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:39:00Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:39:00Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:39:00Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d"} {"level":"info","ts":"2026-09-11T12:39:00Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:39:00Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:39:00Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:39:00Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"console-dashboards","resources":4} {"level":"error","ts":"2026-09-11T12:39:02Z","msg":"component reconciliation failed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","component":"deploy","error":"applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"github.com/opendatahub-io/kserve-module/pkg/kservemodule.(*KserveModuleReconciler).Reconcile\n\t/go/src/github.com/opendatahub-io/kserve-module/pkg/kservemodule/reconciler.go:189\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-09-11T12:39:02Z","msg":"Reconciler error","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1abe83c-d959-4973-a7ab-1094dc20bf9d","error":"reconciliation failed: deploy: applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-09-11T12:39:02Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","name":"default-kserve"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:39:06Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:39:07Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:39:07Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:39:07Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:39:07Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7"} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:39:08Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"console-dashboards","resources":4} {"level":"error","ts":"2026-09-11T12:39:09Z","msg":"component reconciliation failed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","component":"deploy","error":"applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"github.com/opendatahub-io/kserve-module/pkg/kservemodule.(*KserveModuleReconciler).Reconcile\n\t/go/src/github.com/opendatahub-io/kserve-module/pkg/kservemodule/reconciler.go:189\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-09-11T12:39:09Z","msg":"Reconciler error","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3118167e-59df-4f7d-8aaf-cda8e459e1f7","error":"reconciliation failed: deploy: applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-09-11T12:39:09Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","name":"default-kserve"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:39:14Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:39:14Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:39:14Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:39:14Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:39:14Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:39:14Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:39:15Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:39:15Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:39:15Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:39:15Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c"} {"level":"info","ts":"2026-09-11T12:39:15Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:39:15Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:39:15Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:39:15Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"console-dashboards","resources":4} {"level":"error","ts":"2026-09-11T12:39:17Z","msg":"component reconciliation failed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","component":"deploy","error":"applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"github.com/opendatahub-io/kserve-module/pkg/kservemodule.(*KserveModuleReconciler).Reconcile\n\t/go/src/github.com/opendatahub-io/kserve-module/pkg/kservemodule/reconciler.go:189\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"error","ts":"2026-09-11T12:39:17Z","msg":"Reconciler error","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"e1fbd61c-6d9c-4ed2-8312-f27eded94a0c","error":"reconciliation failed: deploy: applying unowned resources: failure deploying resource redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: apply failed serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig: unable to patch serving.kserve.io/v1alpha2, Kind=LLMInferenceServiceConfig redhat-ods-applications/v3-6-0-ea-1-kserve-config-llm-decode-template: Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.redhat-ods-applications.svc:443/validate-serving-kserve-io-v1alpha2-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313"} {"level":"info","ts":"2026-09-11T12:39:17Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","name":"default-kserve"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:39:22Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db"} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:39:23Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","component":"console-dashboards","resources":4} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"deployed all resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"890d388d-6abb-4ee7-b1aa-ae2c083936db","owned":104,"unowned":20} {"level":"info","ts":"2026-09-11T12:39:27Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","name":"default-kserve"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:39:32Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:39:32Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:39:32Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:39:32Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:39:32Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:39:32Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:39:33Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:39:33Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:39:33Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:39:33Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c"} {"level":"info","ts":"2026-09-11T12:39:33Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:39:33Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:39:33Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:39:33Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","component":"console-dashboards","resources":4} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"deployed all resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f21df1fb-a91f-4df5-ae55-66a36c89b16c","owned":104,"unowned":20} {"level":"info","ts":"2026-09-11T12:39:36Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","name":"default-kserve"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:39:41Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:39:41Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:39:41Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:39:41Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:39:41Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:39:41Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:39:42Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:39:42Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:39:42Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:39:42Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b"} {"level":"info","ts":"2026-09-11T12:39:42Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:39:42Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:39:42Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:39:42Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","component":"console-dashboards","resources":4} {"level":"info","ts":"2026-09-11T12:39:45Z","msg":"deployed all resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"77b07a6a-73ee-4203-a7d3-3bdece17e06b","owned":104,"unowned":20} {"level":"info","ts":"2026-09-11T12:39:45Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","name":"default-kserve"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:39:50Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:39:50Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:39:50Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:39:50Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:39:50Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:39:50Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:39:50Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:39:51Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:39:51Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:39:51Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3"} {"level":"info","ts":"2026-09-11T12:39:51Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:39:51Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:39:51Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:39:51Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","component":"console-dashboards","resources":4} {"level":"info","ts":"2026-09-11T12:39:53Z","msg":"deployed all resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"3ab5c18d-ae14-461b-8b1b-49464f8de3a3","owned":104,"unowned":20} {"level":"info","ts":"2026-09-11T12:45:46Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","name":"default-kserve"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:45:51Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:45:51Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:45:51Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:45:51Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:45:51Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:45:51Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:45:52Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:45:52Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:45:52Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:45:52Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9"} {"level":"info","ts":"2026-09-11T12:45:52Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:45:52Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:45:52Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:45:52Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","component":"console-dashboards","resources":4} {"level":"info","ts":"2026-09-11T12:45:54Z","msg":"deployed all resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a00fce87-7e72-49f4-8397-c2f46c04bbf9","owned":104,"unowned":20} {"level":"info","ts":"2026-09-11T12:45:55Z","msg":"reconciling Kserve CR","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","name":"default-kserve"} # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. {"level":"info","ts":"2026-09-11T12:45:59Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"kserve","resourceCount":95} {"level":"info","ts":"2026-09-11T12:45:59Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","before":95,"after":77,"removed":18} {"level":"info","ts":"2026-09-11T12:45:59Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"kserve","resources":77} {"level":"info","ts":"2026-09-11T12:46:00Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"modelcontroller","resourceCount":63} {"level":"info","ts":"2026-09-11T12:46:00Z","msg":"filtered fast-variant resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","before":63,"after":43,"removed":20} {"level":"info","ts":"2026-09-11T12:46:00Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"modelcontroller","resources":43} {"level":"info","ts":"2026-09-11T12:46:00Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"wva","resourceCount":24} {"level":"info","ts":"2026-09-11T12:46:01Z","msg":"default cleanup completed","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"modelcache","resourceCount":28} {"level":"info","ts":"2026-09-11T12:46:01Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"observability","resourceCount":3} {"level":"info","ts":"2026-09-11T12:46:01Z","msg":"PersesDashboard CRD not available, skipping observability dashboards","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520"} {"level":"info","ts":"2026-09-11T12:46:01Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"observability","resources":0} {"level":"info","ts":"2026-09-11T12:46:01Z","msg":"rendered kustomize manifests","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"console-dashboards","resourceCount":4} {"level":"info","ts":"2026-09-11T12:46:01Z","msg":"set namespace on console dashboard resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","namespace":"openshift-config-managed","count":4} {"level":"info","ts":"2026-09-11T12:46:01Z","msg":"component rendering complete","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","component":"console-dashboards","resources":4} {"level":"info","ts":"2026-09-11T12:46:03Z","msg":"deployed all resources","controller":"kserve-module","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"995b95ca-fa6b-49e2-88ba-ec0e9baf8520","owned":104,"unowned":20} ==================== WORKLOAD POD LOG: redhat-ods-applications/kuberay-operator-b5fb5769f-c82q2.log ==================== {"level":"info","ts":"2026-09-11T12:37:50.385Z","logger":"setup","msg":"Loaded feature gates","featureGates":{"AsyncJobInfoQuery":false,"RayClusterStatusConditions":true,"RayCronJob":false,"RayJobDeletionPolicy":true,"RayMultiHostIndexing":true,"RayServiceIncrementalUpgrade":false}} {"level":"info","ts":"2026-09-11T12:37:50.385Z","logger":"setup","msg":"Flag watchNamespace is not set. Watch custom resources in all namespaces."} {"level":"info","ts":"2026-09-11T12:37:50.385Z","logger":"setup","msg":"Setup manager"} {"level":"info","ts":"2026-09-11T12:37:50.782Z","logger":"tls","msg":"TLS profile watcher registered"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"setup","msg":"NetworkPolicy controller registered (annotation-based activation)"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"setup","msg":"RayCronJob feature gate is disabled, skipping RayCronJob controller setup"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"setup","msg":"starting manager"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","msg":"starting server","name":"health probe","addr":"[::]:8082"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/etc/metrics-certs/tls.crt","key":"/etc/metrics-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:50.883Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-09-11T12:37:50.884Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":true} {"level":"info","ts":"2026-09-11T12:37:50.884Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-09-11T12:37:50.884Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/etc/metrics-certs/tls.crt","key":"/etc/metrics-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:50.884Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"info","ts":"2026-09-11T12:37:51.184Z","msg":"Starting EventSource","controller":"tls-profile-watcher","source":"kind source: *v1.APIServer"} {"level":"info","ts":"2026-09-11T12:37:51.184Z","msg":"Attempting to acquire leader lease...","lock":"redhat-ods-applications/ray-operator-leader"} {"level":"info","ts":"2026-09-11T12:37:51.280Z","msg":"Successfully acquired lease","lock":"redhat-ods-applications/ray-operator-leader"} {"level":"info","ts":"2026-09-11T12:37:51.280Z","logger":"controllers.RayCluster","msg":"Starting EventSource","source":"kind source: *v1.Secret"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayCluster","msg":"Starting EventSource","source":"kind source: *v1.RayCluster"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayJob","msg":"Starting EventSource","source":"kind source: *v1.Job"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayCluster","msg":"Starting EventSource","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayCluster","msg":"Starting EventSource","source":"kind source: *v1.Service"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayJob","msg":"Starting EventSource","source":"kind source: *v1.RayJob"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayJob","msg":"Starting EventSource","source":"kind source: *v1.RayCluster"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayJob","msg":"Starting EventSource","source":"kind source: *v1.Service"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayService","msg":"Starting EventSource","source":"kind source: *v1.Service"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","msg":"Starting EventSource","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","source":"kind source: *v1.NetworkPolicy"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayService","msg":"Starting EventSource","source":"kind source: *v1.RayService"} {"level":"info","ts":"2026-09-11T12:37:51.281Z","logger":"controllers.RayService","msg":"Starting EventSource","source":"kind source: *v1.RayCluster"} {"level":"info","ts":"2026-09-11T12:37:51.282Z","msg":"Starting EventSource","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","source":"kind source: *v1.RayCluster"} {"level":"info","ts":"2026-09-11T12:37:51.282Z","msg":"Starting EventSource","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","source":"kind source: *v1.RayCluster"} {"level":"info","ts":"2026-09-11T12:37:51.282Z","msg":"Starting EventSource","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","source":"kind source: *v1.Route"} {"level":"info","ts":"2026-09-11T12:37:51.282Z","msg":"Starting EventSource","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","source":"kind source: *v1.ServiceAccount"} {"level":"info","ts":"2026-09-11T12:37:51.282Z","msg":"Starting EventSource","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","source":"kind source: *v1.Service"} {"level":"info","ts":"2026-09-11T12:37:51.282Z","msg":"Starting EventSource","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","source":"kind source: *v1.RayCluster"} {"level":"info","ts":"2026-09-11T12:37:52.084Z","logger":"controllers.RayCluster","msg":"Starting Controller"} {"level":"info","ts":"2026-09-11T12:37:52.084Z","logger":"controllers.RayCluster","msg":"Starting workers","worker count":1} {"level":"info","ts":"2026-09-11T12:37:52.084Z","msg":"Starting Controller","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster"} {"level":"info","ts":"2026-09-11T12:37:52.084Z","msg":"Starting workers","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","worker count":1} {"level":"info","ts":"2026-09-11T12:37:52.084Z","logger":"controllers.RayService","msg":"Starting Controller"} {"level":"info","ts":"2026-09-11T12:37:52.084Z","logger":"controllers.RayService","msg":"Starting workers","worker count":1} {"level":"info","ts":"2026-09-11T12:37:52.084Z","msg":"Starting Controller","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster"} {"level":"info","ts":"2026-09-11T12:37:52.084Z","msg":"Starting workers","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","worker count":1} {"level":"info","ts":"2026-09-11T12:37:52.085Z","msg":"Starting Controller","controller":"tls-profile-watcher"} {"level":"info","ts":"2026-09-11T12:37:52.085Z","msg":"Starting workers","controller":"tls-profile-watcher","worker count":1} {"level":"info","ts":"2026-09-11T12:37:52.085Z","msg":"Starting Controller","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster"} {"level":"info","ts":"2026-09-11T12:37:52.085Z","msg":"Starting workers","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","worker count":1} {"level":"info","ts":"2026-09-11T12:37:52.101Z","logger":"controllers.RayJob","msg":"Starting Controller"} {"level":"info","ts":"2026-09-11T12:37:52.101Z","logger":"controllers.RayJob","msg":"Starting workers","worker count":1} {"level":"info","ts":"2026-09-11T12:58:22.489Z","logger":"raycluster-resource","msg":"default","name":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.489Z","logger":"raycluster-resource","msg":"enforcing secure trusted network on OpenShift","name":"raycluster-auth-token","namespace":"test-ns-r7tdx"} {"level":"info","ts":"2026-09-11T12:58:22.514Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T12:58:22.514Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.514Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.514Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"816d829e-685b-4db7-a942-df8a8925cf93","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.514Z","logger":"authentication-controller","msg":"Adding authentication finalizer to RayCluster","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.515Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.515Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b"} {"level":"info","ts":"2026-09-11T12:58:22.515Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b"} {"level":"info","ts":"2026-09-11T12:58:22.515Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b"} {"level":"info","ts":"2026-09-11T12:58:22.529Z","logger":"raycluster-resource","msg":"default","name":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.529Z","logger":"raycluster-resource","msg":"enforcing secure trusted network on OpenShift","name":"raycluster-auth-token","namespace":"test-ns-r7tdx"} {"level":"info","ts":"2026-09-11T12:58:22.580Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"816d829e-685b-4db7-a942-df8a8925cf93","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.580Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0519c65c-140f-4aa5-b15e-262bd9eee528","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.598Z","logger":"authentication-controller","msg":"Service account reconciled","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","name":"raycluster-auth-token-oauth-proxy-sa","operation":"created","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T12:58:22.599Z","logger":"controllers.RayCluster","msg":"Created service for RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","name":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:22.599Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","error":"head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"error","ts":"2026-09-11T12:58:22.599Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.599Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"info","ts":"2026-09-11T12:58:22.599Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.599Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.599Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.599Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","name":"raycluster-auth-token","old status":{"desiredCPU":"0","desiredMemory":"0","desiredGPU":"0","desiredTPU":"0","head":{}},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T12:58:22Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"Error updating status","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","name":"raycluster-auth-token","error":"Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-auth-token\": the object has been modified; please apply your changes to the latest version and try again","RayCluster":{"apiVersion":"ray.io/v1","kind":"RayCluster","namespace":"test-ns-r7tdx","name":"raycluster-auth-token"}} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b"} {"level":"error","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9f54658a-c5db-463e-85bb-3851babd806b","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.615Z","msg":"Creating self-signed issuer for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2"} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2"} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2"} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2","error":"head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"error","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.615Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.616Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.616Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2","name":"raycluster-auth-token","old status":{"desiredCPU":"0","desiredMemory":"0","desiredGPU":"0","desiredTPU":"0","head":{}},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T12:58:22Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T12:58:22.618Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0519c65c-140f-4aa5-b15e-262bd9eee528","name":"raycluster-auth-token-head"} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2"} {"level":"error","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a1addd24-6bc8-41c6-b887-5cb6676b82e2","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245"} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245"} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245"} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","error":"head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"error","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.637Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","name":"raycluster-auth-token","old status":{"desiredCPU":"0","desiredMemory":"0","desiredGPU":"0","desiredTPU":"0","head":{}},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T12:58:22Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T12:58:22.651Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0519c65c-140f-4aa5-b15e-262bd9eee528","name":"raycluster-auth-token-workers"} {"level":"info","ts":"2026-09-11T12:58:22.651Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0519c65c-140f-4aa5-b15e-262bd9eee528","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.651Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"cd7e8579-107e-4c1d-b3f1-06ebcdcaeb8e","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.654Z","logger":"controllers.RayCluster","msg":"Error updating status","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","name":"raycluster-auth-token","error":"Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-auth-token\": the object has been modified; please apply your changes to the latest version and try again","RayCluster":{"apiVersion":"ray.io/v1","kind":"RayCluster","namespace":"test-ns-r7tdx","name":"raycluster-auth-token"}} {"level":"info","ts":"2026-09-11T12:58:22.654Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245"} {"level":"error","ts":"2026-09-11T12:58:22.654Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9e80aedc-74d1-4bcd-8c15-e2da84b83245","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.654Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260"} {"level":"info","ts":"2026-09-11T12:58:22.654Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260"} {"level":"info","ts":"2026-09-11T12:58:22.654Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260"} {"level":"info","ts":"2026-09-11T12:58:22.654Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:22.654Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260","error":"head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"error","ts":"2026-09-11T12:58:22.654Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.655Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"info","ts":"2026-09-11T12:58:22.655Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.655Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.655Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.655Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260"} {"level":"error","ts":"2026-09-11T12:58:22.655Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"756420a8-15e3-42c5-b3da-d126d4630260","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1"} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1"} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1"} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1","error":"head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"error","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1"} {"level":"error","ts":"2026-09-11T12:58:22.680Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"10721764-bfe4-4d3a-8158-6181d19a1bf1","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.682Z","msg":"Self-signed issuer created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.695Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"cd7e8579-107e-4c1d-b3f1-06ebcdcaeb8e","name":"raycluster-auth-token-head"} {"level":"info","ts":"2026-09-11T12:58:22.715Z","logger":"authentication-controller","msg":"ReferenceGrant reconciled (shared)","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","name":"kuberay-gateway-access","namespace":"test-ns-r7tdx","operation":"created"} {"level":"info","ts":"2026-09-11T12:58:22.728Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"cd7e8579-107e-4c1d-b3f1-06ebcdcaeb8e","name":"raycluster-auth-token-workers"} {"level":"info","ts":"2026-09-11T12:58:22.728Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"cd7e8579-107e-4c1d-b3f1-06ebcdcaeb8e","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.761Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146"} {"level":"info","ts":"2026-09-11T12:58:22.761Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146"} {"level":"info","ts":"2026-09-11T12:58:22.761Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146"} {"level":"info","ts":"2026-09-11T12:58:22.761Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:22.761Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146","error":"head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"error","ts":"2026-09-11T12:58:22.761Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.761Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"info","ts":"2026-09-11T12:58:22.762Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.762Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.762Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.762Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146"} {"level":"error","ts":"2026-09-11T12:58:22.762Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"708d80a5-dd94-4cad-965c-259d61c65146","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.782Z","msg":"Creating CA certificate for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.817Z","msg":"CA Certificate created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","rayCluster":"raycluster-auth-token","secret":"ca-secret"} {"level":"info","ts":"2026-09-11T12:58:22.817Z","msg":"Creating CA issuer for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.839Z","logger":"authentication-controller","msg":"HTTPRoute reconciled","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","name":"test-ns-r7tdx-raycluster-auth-token","operation":"created"} {"level":"info","ts":"2026-09-11T12:58:22.850Z","msg":"CA issuer created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d"} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d"} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d"} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d","error":"head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"error","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d"} {"level":"error","ts":"2026-09-11T12:58:22.985Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"2906a62e-df66-4fe0-99bf-bfb31c08d29d","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.182Z","msg":"Ray head certificate created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","rayCluster":"raycluster-auth-token","podIPs":[]} {"level":"info","ts":"2026-09-11T12:58:23.283Z","msg":"Ray worker certificate created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","rayCluster":"raycluster-auth-token","podIPs":[]} {"level":"error","ts":"2026-09-11T12:58:23.283Z","msg":"Failed to check certificate readiness","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","error":"Certificate.cert-manager.io \"ray-worker-cert-raycluster-auth-token\" not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:220\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.283Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4"} {"level":"error","ts":"2026-09-11T12:58:23.283Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3deba3ee-8c90-4c49-a40a-d267a49b02e4","error":"Certificate.cert-manager.io \"ray-worker-cert-raycluster-auth-token\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.283Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"7369da45-f016-4d16-944f-bac6f1c8f425","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e"} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e"} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e"} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e","error":"head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"error","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found"} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e"} {"level":"error","ts":"2026-09-11T12:58:23.380Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"51b668bb-ab5e-4ff1-906b-82e83964293e","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-auth-token not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T12:58:23.480Z","msg":"Failed to create Ray worker certificate","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"7369da45-f016-4d16-944f-bac6f1c8f425","error":"certificates.cert-manager.io \"ray-worker-cert-raycluster-auth-token\" already exists","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).createRayWorkerCertificate\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:732\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:206\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T12:58:23.481Z","msg":"Failed to create Ray worker certificate","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"7369da45-f016-4d16-944f-bac6f1c8f425","error":"certificates.cert-manager.io \"ray-worker-cert-raycluster-auth-token\" already exists","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:207\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.481Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"7369da45-f016-4d16-944f-bac6f1c8f425"} {"level":"error","ts":"2026-09-11T12:58:23.481Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"7369da45-f016-4d16-944f-bac6f1c8f425","error":"certificates.cert-manager.io \"ray-worker-cert-raycluster-auth-token\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.481Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"44155a8b-2df5-496e-b9e1-03725c6acdc2","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.481Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"44155a8b-2df5-496e-b9e1-03725c6acdc2"} {"level":"info","ts":"2026-09-11T12:58:23.581Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"637e392b-c5af-41ab-9b2d-378b2b00d16c","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.581Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"637e392b-c5af-41ab-9b2d-378b2b00d16c"} {"level":"info","ts":"2026-09-11T12:58:23.755Z","logger":"authentication-controller","msg":"OIDC ConfigMap reconciled","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","name":"kube-rbac-proxy-config-raycluster-auth-token","operation":"created"} {"level":"error","ts":"2026-09-11T12:58:23.771Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","error":"authentication resources created but failed to update AuthenticationReady condition: Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-auth-token\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.771Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8"} {"level":"error","ts":"2026-09-11T12:58:23.771Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"94415d01-19e3-4b49-92d4-489b0e38e7d8","error":"authentication resources created but failed to update AuthenticationReady condition: Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-auth-token\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.771Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"1954b6b9-66b2-4d96-984b-55b906390cc0","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T12:58:23.771Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"1954b6b9-66b2-4d96-984b-55b906390cc0","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.771Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"1954b6b9-66b2-4d96-984b-55b906390cc0","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.793Z","logger":"authentication-controller","msg":"Successfully reconciled authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"1954b6b9-66b2-4d96-984b-55b906390cc0","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.793Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"80045428-abb7-4f10-950e-bb5a1b5b4a4f","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T12:58:23.793Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"80045428-abb7-4f10-950e-bb5a1b5b4a4f","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.793Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"80045428-abb7-4f10-950e-bb5a1b5b4a4f","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.793Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"310676f6-366d-4def-a0c2-3771dca64bbf","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.794Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"5b5d4726-781b-4497-9c9b-b5b980fa4fd4","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.794Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"5b5d4726-781b-4497-9c9b-b5b980fa4fd4"} {"level":"error","ts":"2026-09-11T12:58:23.813Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"80045428-abb7-4f10-950e-bb5a1b5b4a4f","error":"authentication resources created but failed to update AuthenticationReady condition: Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-auth-token\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.813Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"80045428-abb7-4f10-950e-bb5a1b5b4a4f"} {"level":"error","ts":"2026-09-11T12:58:23.813Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"80045428-abb7-4f10-950e-bb5a1b5b4a4f","error":"authentication resources created but failed to update AuthenticationReady condition: Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-auth-token\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:23.818Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4e22dd85-cf0a-4f7b-93db-9b187f77c8e7","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T12:58:23.818Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4e22dd85-cf0a-4f7b-93db-9b187f77c8e7","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.818Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4e22dd85-cf0a-4f7b-93db-9b187f77c8e7","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.827Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"310676f6-366d-4def-a0c2-3771dca64bbf","name":"raycluster-auth-token-head"} {"level":"info","ts":"2026-09-11T12:58:23.837Z","logger":"authentication-controller","msg":"Successfully reconciled authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4e22dd85-cf0a-4f7b-93db-9b187f77c8e7","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:23.861Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"310676f6-366d-4def-a0c2-3771dca64bbf","name":"raycluster-auth-token-workers"} {"level":"info","ts":"2026-09-11T12:58:23.861Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"310676f6-366d-4def-a0c2-3771dca64bbf","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2"} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2"} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2"} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2","error":"worker certificate secret ray-worker-secret-raycluster-auth-token not found"} {"level":"error","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: worker certificate secret ray-worker-secret-raycluster-auth-token not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2","error":"mTLS secrets not ready: worker certificate secret ray-worker-secret-raycluster-auth-token not found"} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2"} {"level":"error","ts":"2026-09-11T12:58:24.021Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3b5ad1be-0749-43fd-b1c5-c48fbb7f3aa2","error":"mTLS secrets not ready: worker certificate secret ray-worker-secret-raycluster-auth-token not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","cluster":"raycluster-auth-token","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","cluster":"raycluster-auth-token","authType":"IntegratedOAuth","serviceAccount":"raycluster-auth-token-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-auth-token","ray.io/group":"headgroup","ray.io/identifier":"raycluster-auth-token-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T12:58:25.302Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","enabled":true} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"Created head Pod for RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","name":"raycluster-auth-token-head-zm8sh"} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a"} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T12:58:25.424Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","enabled":true} {"level":"info","ts":"2026-09-11T12:58:25.509Z","logger":"controllers.RayCluster","msg":"Created worker Pod for RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","name":"raycluster-auth-token-small-group-worker-stx4v"} {"level":"info","ts":"2026-09-11T12:58:25.510Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.510Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.510Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"11246e56-9591-4dd7-b929-d63cf0a3ce5a","name":"raycluster-auth-token","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T12:58:22Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T12:58:25Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4559473f-2197-421f-88ee-6575c377ed73","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"93228553-c28b-4671-b6f6-b04debd259c0","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:25.531Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","name":"raycluster-auth-token","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T12:58:22Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T12:58:25Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T12:58:25.531Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"93228553-c28b-4671-b6f6-b04debd259c0","rayCluster":"raycluster-auth-token","podIPs":[]} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"Error updating status","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","name":"raycluster-auth-token","error":"Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-auth-token\": the object has been modified; please apply your changes to the latest version and try again","RayCluster":{"apiVersion":"ray.io/v1","kind":"RayCluster","namespace":"test-ns-r7tdx","name":"raycluster-auth-token"}} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1"} {"level":"error","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"14b95553-d6b7-472b-bf10-b1dfc1a6d8f1","error":"Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-auth-token\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a"} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a"} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a"} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.550Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.551Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T12:58:25.551Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f05762bf-13a4-4331-8230-934a4245c66a","seconds":300} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435"} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435"} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435"} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T12:58:25.556Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"561edb14-2400-4a56-8c3c-2a313d9c3435","seconds":300} {"level":"info","ts":"2026-09-11T12:58:25.565Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4559473f-2197-421f-88ee-6575c377ed73","name":"raycluster-auth-token-head"} {"level":"info","ts":"2026-09-11T12:58:25.566Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1"} {"level":"info","ts":"2026-09-11T12:58:25.566Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1"} {"level":"info","ts":"2026-09-11T12:58:25.566Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1"} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T12:58:25.567Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f0f810a3-b982-48ae-be02-510acce0a2e1","seconds":300} {"level":"info","ts":"2026-09-11T12:58:25.599Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4559473f-2197-421f-88ee-6575c377ed73","name":"raycluster-auth-token-workers"} {"level":"info","ts":"2026-09-11T12:58:25.599Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4559473f-2197-421f-88ee-6575c377ed73","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:26.064Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab"} {"level":"info","ts":"2026-09-11T12:58:26.064Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab"} {"level":"info","ts":"2026-09-11T12:58:26.064Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab"} {"level":"info","ts":"2026-09-11T12:58:26.064Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T12:58:26.065Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ee4b9254-8dd0-453b-86e1-486fe31989ab","seconds":300} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41"} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41"} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41"} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:26.159Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T12:58:26.160Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T12:58:26.160Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3650e3a8-e664-4df3-9190-4015683ddf41","seconds":300} {"level":"info","ts":"2026-09-11T12:58:53.582Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"9354c1ff-af9c-45f6-911f-1a8154a27789","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:58:53.582Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"9354c1ff-af9c-45f6-911f-1a8154a27789","rayCluster":"raycluster-auth-token","podIPs":[]} {"level":"info","ts":"2026-09-11T12:59:53.582Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"01ad0bf0-bfbc-47d8-8faa-233e388b24f9","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T12:59:53.583Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"01ad0bf0-bfbc-47d8-8faa-233e388b24f9","rayCluster":"raycluster-auth-token","podIPs":[]} {"level":"info","ts":"2026-09-11T13:00:53.584Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"434b2aef-843a-464b-a294-1660e5a98563","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:00:53.584Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"434b2aef-843a-464b-a294-1660e5a98563","rayCluster":"raycluster-auth-token","podIPs":[]} {"level":"info","ts":"2026-09-11T13:00:59.403Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d"} {"level":"info","ts":"2026-09-11T13:00:59.403Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d"} {"level":"info","ts":"2026-09-11T13:00:59.403Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d"} {"level":"info","ts":"2026-09-11T13:00:59.403Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:00:59.404Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"865166fa-87e7-4a90-a592-988e11d1d51d","name":"raycluster-auth-token","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T12:58:25Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:00:59Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:00:59.425Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c86fffd1-f5e0-4ed3-8c79-220e9cc3d27d","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:00:59.425Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"30b5cf30-feeb-4c57-885e-67286e72bbff","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:00:59.426Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c86fffd1-f5e0-4ed3-8c79-220e9cc3d27d","pod":"raycluster-auth-token-head-zm8sh","ip":"10.134.0.43"} {"level":"info","ts":"2026-09-11T13:00:59.464Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"30b5cf30-feeb-4c57-885e-67286e72bbff","name":"raycluster-auth-token-head"} {"level":"info","ts":"2026-09-11T13:00:59.497Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"30b5cf30-feeb-4c57-885e-67286e72bbff","name":"raycluster-auth-token-workers"} {"level":"info","ts":"2026-09-11T13:00:59.497Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"30b5cf30-feeb-4c57-885e-67286e72bbff","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:00:59.498Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c86fffd1-f5e0-4ed3-8c79-220e9cc3d27d","certificate":"ray-head-cert-raycluster-auth-token","dnsNames":["raycluster-auth-token-head-svc","localhost","raycluster-auth-token-head-svc.test-ns-r7tdx.svc","raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local"],"ipAddresses":["10.134.0.43","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:00:59.532Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c86fffd1-f5e0-4ed3-8c79-220e9cc3d27d","certificate":"ray-worker-cert-raycluster-auth-token","dnsNames":["raycluster-auth-token-worker-svc","localhost","raycluster-auth-token-small-group","raycluster-auth-token-worker-svc.test-ns-r7tdx.svc","raycluster-auth-token-worker-svc.test-ns-r7tdx.svc.cluster.local","*.raycluster-auth-token-worker-svc.test-ns-r7tdx.svc","*.raycluster-auth-token-worker-svc.test-ns-r7tdx.svc.cluster.local","*-worker-*.test-ns-r7tdx.svc","*-worker-*.test-ns-r7tdx.svc.cluster.local"],"ipAddresses":["10.134.0.43","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:00:59.532Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c86fffd1-f5e0-4ed3-8c79-220e9cc3d27d","rayCluster":"raycluster-auth-token","podIPs":["10.134.0.43"]} {"level":"info","ts":"2026-09-11T13:01:01.426Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2"} {"level":"info","ts":"2026-09-11T13:01:01.426Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2"} {"level":"info","ts":"2026-09-11T13:01:01.426Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2"} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:01:01.427Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"a383b69e-4f23-43a1-aa2e-720bd40ed3f2","seconds":300} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991"} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991"} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991"} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:01:02.017Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"58b78591-5c8f-4010-b6c7-d9cbf76ee991","seconds":300} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4"} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4"} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4"} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","podName":"raycluster-auth-token-head-zm8sh","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:01:05.437Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7f061273-a001-4264-827e-61984878c4e4","seconds":300} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe"} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe"} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe"} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:01:08.478Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:01:08.479Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not ready.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","podName":"raycluster-auth-token-head-zm8sh","pod Status.Conditions[PodReady]":{"type":"Ready","observedGeneration":1,"status":"False","lastProbeTime":null,"lastTransitionTime":"2026-09-11T12:58:25Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-head]"}} {"level":"info","ts":"2026-09-11T13:01:08.479Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not ready.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","podName":"raycluster-auth-token-head-zm8sh","pod Status.Conditions[PodReady]":{"type":"Ready","observedGeneration":1,"status":"False","lastProbeTime":null,"lastTransitionTime":"2026-09-11T12:58:25Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-head]"}} {"level":"info","ts":"2026-09-11T13:01:08.479Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b95d3292-70ae-4316-8c7d-8bfe473f9bbe","name":"raycluster-auth-token","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:00:59Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:01:08Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-head]"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:01:08.500Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"d5f3d646-b4a4-429b-8e25-5fff1247afcf","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:01:08.500Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c0f14de3-5653-4702-8f13-85634cf98462","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:01:08.500Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"d5f3d646-b4a4-429b-8e25-5fff1247afcf","pod":"raycluster-auth-token-head-zm8sh","ip":"10.134.0.43"} {"level":"info","ts":"2026-09-11T13:01:08.500Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"d5f3d646-b4a4-429b-8e25-5fff1247afcf","pod":"raycluster-auth-token-small-group-worker-stx4v","ip":"10.133.0.63"} {"level":"info","ts":"2026-09-11T13:01:08.533Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"d5f3d646-b4a4-429b-8e25-5fff1247afcf","certificate":"ray-head-cert-raycluster-auth-token","dnsNames":["raycluster-auth-token-head-svc","localhost","raycluster-auth-token-head-svc.test-ns-r7tdx.svc","raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local"],"ipAddresses":["10.134.0.43","10.133.0.63","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:01:08.534Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c0f14de3-5653-4702-8f13-85634cf98462","name":"raycluster-auth-token-head"} {"level":"info","ts":"2026-09-11T13:01:08.566Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"d5f3d646-b4a4-429b-8e25-5fff1247afcf","certificate":"ray-worker-cert-raycluster-auth-token","dnsNames":["raycluster-auth-token-worker-svc","localhost","raycluster-auth-token-small-group","raycluster-auth-token-worker-svc.test-ns-r7tdx.svc","raycluster-auth-token-worker-svc.test-ns-r7tdx.svc.cluster.local","*.raycluster-auth-token-worker-svc.test-ns-r7tdx.svc","*.raycluster-auth-token-worker-svc.test-ns-r7tdx.svc.cluster.local","*-worker-*.test-ns-r7tdx.svc","*-worker-*.test-ns-r7tdx.svc.cluster.local"],"ipAddresses":["10.134.0.43","10.133.0.63","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:01:08.566Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"d5f3d646-b4a4-429b-8e25-5fff1247afcf","rayCluster":"raycluster-auth-token","podIPs":["10.134.0.43","10.133.0.63"]} {"level":"info","ts":"2026-09-11T13:01:08.568Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c0f14de3-5653-4702-8f13-85634cf98462","name":"raycluster-auth-token-workers"} {"level":"info","ts":"2026-09-11T13:01:08.568Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c0f14de3-5653-4702-8f13-85634cf98462","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6"} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6"} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6"} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:01:10.500Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:01:10.501Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not ready.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","podName":"raycluster-auth-token-head-zm8sh","pod Status.Conditions[PodReady]":{"type":"Ready","observedGeneration":1,"status":"False","lastProbeTime":null,"lastTransitionTime":"2026-09-11T12:58:25Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-head]"}} {"level":"info","ts":"2026-09-11T13:01:10.501Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not ready.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","podName":"raycluster-auth-token-head-zm8sh","pod Status.Conditions[PodReady]":{"type":"Ready","observedGeneration":1,"status":"False","lastProbeTime":null,"lastTransitionTime":"2026-09-11T12:58:25Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-head]"}} {"level":"info","ts":"2026-09-11T13:01:10.501Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:01:10.501Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac000450-86ec-4fe4-b3d7-8461517bf8a6","seconds":300} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570"} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570"} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570"} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:19.514Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9ca3ef84-97ac-48fc-a0a8-e0878d505570","name":"raycluster-auth-token","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:01:08Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-head]"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:01:19Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:01:19.536Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4498958d-52e1-42b0-b955-fcb0bd766149","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:01:19.536Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"746316d6-c599-486b-955e-954185963204","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:01:19.536Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4498958d-52e1-42b0-b955-fcb0bd766149","pod":"raycluster-auth-token-head-zm8sh","ip":"10.134.0.43"} {"level":"info","ts":"2026-09-11T13:01:19.536Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4498958d-52e1-42b0-b955-fcb0bd766149","pod":"raycluster-auth-token-small-group-worker-stx4v","ip":"10.133.0.63"} {"level":"info","ts":"2026-09-11T13:01:19.536Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4498958d-52e1-42b0-b955-fcb0bd766149","rayCluster":"raycluster-auth-token","podIPs":["10.134.0.43","10.133.0.63"]} {"level":"info","ts":"2026-09-11T13:01:19.576Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"746316d6-c599-486b-955e-954185963204","name":"raycluster-auth-token-head"} {"level":"info","ts":"2026-09-11T13:01:19.607Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"746316d6-c599-486b-955e-954185963204","name":"raycluster-auth-token-workers"} {"level":"info","ts":"2026-09-11T13:01:19.607Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"746316d6-c599-486b-955e-954185963204","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926"} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926"} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926"} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:21.537Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:01:21.538Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:01:21.538Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9999d096-1a7a-45af-a8fe-eabb1f5ff926","seconds":300} {"level":"info","ts":"2026-09-11T13:01:53.584Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4288ee1b-bb5c-4bd3-95e8-abfa544cd6c6","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:01:53.585Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4288ee1b-bb5c-4bd3-95e8-abfa544cd6c6","pod":"raycluster-auth-token-head-zm8sh","ip":"10.134.0.43"} {"level":"info","ts":"2026-09-11T13:01:53.585Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4288ee1b-bb5c-4bd3-95e8-abfa544cd6c6","pod":"raycluster-auth-token-small-group-worker-stx4v","ip":"10.133.0.63"} {"level":"info","ts":"2026-09-11T13:01:53.585Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4288ee1b-bb5c-4bd3-95e8-abfa544cd6c6","rayCluster":"raycluster-auth-token","podIPs":["10.134.0.43","10.133.0.63"]} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0"} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0"} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0"} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:07.439Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:07.440Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:07.440Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:07.440Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"84284c55-c2f6-4728-93ff-1d3981bf52c0","seconds":300} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79"} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79"} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79"} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:08.448Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:08.449Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:08.449Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:08.449Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:08.449Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"54739b7b-d547-4084-b8ed-dfbc13fc2e79","seconds":300} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d"} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d"} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d"} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:18.517Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:18.518Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:18.518Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:18.518Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:18.518Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"986fff56-cc38-4f85-8284-52e24b51d74d","seconds":300} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71"} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71"} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71"} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not ready.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Conditions[PodReady]":{"type":"Ready","observedGeneration":1,"status":"False","lastProbeTime":null,"lastTransitionTime":"2026-09-11T12:58:25Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-worker]"}} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not ready.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Conditions[PodReady]":{"type":"Ready","observedGeneration":1,"status":"False","lastProbeTime":null,"lastTransitionTime":"2026-09-11T12:58:25Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-worker]"}} {"level":"info","ts":"2026-09-11T13:02:19.526Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"65063e3c-703b-4956-8f7b-b997eb051a71","name":"raycluster-auth-token","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:01:19Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:19Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"availableWorkerReplicas":1,"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:19.548Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"eb374b3a-bb85-4771-b3b7-a046abfb3fd2","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:19.548Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"90f22783-b85b-451f-bb9e-4caf8d596ea6","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:19.548Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"eb374b3a-bb85-4771-b3b7-a046abfb3fd2","pod":"raycluster-auth-token-head-zm8sh","ip":"10.134.0.43"} {"level":"info","ts":"2026-09-11T13:02:19.548Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"eb374b3a-bb85-4771-b3b7-a046abfb3fd2","pod":"raycluster-auth-token-small-group-worker-stx4v","ip":"10.133.0.63"} {"level":"info","ts":"2026-09-11T13:02:19.549Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"eb374b3a-bb85-4771-b3b7-a046abfb3fd2","rayCluster":"raycluster-auth-token","podIPs":["10.134.0.43","10.133.0.63"]} {"level":"info","ts":"2026-09-11T13:02:19.586Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"90f22783-b85b-451f-bb9e-4caf8d596ea6","name":"raycluster-auth-token-head"} {"level":"info","ts":"2026-09-11T13:02:19.618Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"90f22783-b85b-451f-bb9e-4caf8d596ea6","name":"raycluster-auth-token-workers"} {"level":"info","ts":"2026-09-11T13:02:19.618Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"90f22783-b85b-451f-bb9e-4caf8d596ea6","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:21.548Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f"} {"level":"info","ts":"2026-09-11T13:02:21.548Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f"} {"level":"info","ts":"2026-09-11T13:02:21.548Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f"} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not ready.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Conditions[PodReady]":{"type":"Ready","observedGeneration":1,"status":"False","lastProbeTime":null,"lastTransitionTime":"2026-09-11T12:58:25Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-worker]"}} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not ready.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Conditions[PodReady]":{"type":"Ready","observedGeneration":1,"status":"False","lastProbeTime":null,"lastTransitionTime":"2026-09-11T12:58:25Z","reason":"ContainersNotReady","message":"containers with unready status: [ray-worker]"}} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:21.549Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"140d17e7-9e44-4312-afe6-b9ac22107d5f","seconds":300} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541"} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541"} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541"} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:30.564Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8f2d4530-e2b6-4594-b978-73336727a541","name":"raycluster-auth-token","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:19Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"availableWorkerReplicas":1,"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:30Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"readyWorkerReplicas":1,"availableWorkerReplicas":1,"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:30.585Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"8db25826-6080-4031-bc86-6025b4897265","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:30.585Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"ae422b3a-3930-47c0-95f5-9c905f9f8d69","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:30.586Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"ae422b3a-3930-47c0-95f5-9c905f9f8d69","pod":"raycluster-auth-token-head-zm8sh","ip":"10.134.0.43"} {"level":"info","ts":"2026-09-11T13:02:30.586Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"ae422b3a-3930-47c0-95f5-9c905f9f8d69","pod":"raycluster-auth-token-small-group-worker-stx4v","ip":"10.133.0.63"} {"level":"info","ts":"2026-09-11T13:02:30.586Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"ae422b3a-3930-47c0-95f5-9c905f9f8d69","rayCluster":"raycluster-auth-token","podIPs":["10.134.0.43","10.133.0.63"]} {"level":"info","ts":"2026-09-11T13:02:30.616Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"8db25826-6080-4031-bc86-6025b4897265","name":"raycluster-auth-token-head"} {"level":"info","ts":"2026-09-11T13:02:30.648Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"8db25826-6080-4031-bc86-6025b4897265","name":"raycluster-auth-token-workers"} {"level":"info","ts":"2026-09-11T13:02:30.648Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"8db25826-6080-4031-bc86-6025b4897265","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:32.585Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182"} {"level":"info","ts":"2026-09-11T13:02:32.585Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182"} {"level":"info","ts":"2026-09-11T13:02:32.585Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182"} {"level":"info","ts":"2026-09-11T13:02:32.585Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:32.586Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b41c808e-8e86-49ba-b261-6267b16cb182","seconds":300} {"level":"info","ts":"2026-09-11T13:02:42.914Z","logger":"raycluster-resource","msg":"default","name":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:42.914Z","logger":"raycluster-resource","msg":"enforcing secure trusted network on OpenShift","name":"raycluster-image-validation","namespace":"test-ns-kppl8"} {"level":"info","ts":"2026-09-11T13:02:42.966Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:02:42.966Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:42.966Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:42.966Z","logger":"authentication-controller","msg":"Adding authentication finalizer to RayCluster","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:42.966Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"38cc399c-c456-40d8-9c11-24cd9a7f2a5d","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:42.966Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8"} {"level":"info","ts":"2026-09-11T13:02:42.966Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8"} {"level":"info","ts":"2026-09-11T13:02:42.966Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8"} {"level":"info","ts":"2026-09-11T13:02:42.966Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:42.966Z","msg":"Creating self-signed issuer for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:42.982Z","logger":"raycluster-resource","msg":"default","name":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:42.982Z","logger":"raycluster-resource","msg":"enforcing secure trusted network on OpenShift","name":"raycluster-image-validation","namespace":"test-ns-kppl8"} {"level":"info","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"Created service for RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8","name":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8","error":"head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"error","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"info","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"updateEndpoints: Unable to find a Service for this RayCluster. Not adding RayCluster status.endpoints","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8","serviceSelectors":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/identifier":"raycluster-image-validation-head","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"Got error when calculating new status","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8","error":"unable to find head service. cluster name raycluster-image-validation, filter labels [test-ns-kppl8 map[ray.io/cluster:raycluster-image-validation ray.io/identifier:raycluster-image-validation-head ray.io/node-type:head]]"} {"level":"info","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8"} {"level":"error","ts":"2026-09-11T13:02:42.984Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f8cc5511-08c4-4f53-b077-beb23da62cd8","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d"} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d"} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d"} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","error":"head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"error","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:42.985Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","name":"raycluster-image-validation","old status":{"desiredCPU":"0","desiredMemory":"0","desiredGPU":"0","desiredTPU":"0","head":{}},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:42Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:42Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:42Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:42Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:42Z","reason":"RayClusterSuspending","message":""}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:43.003Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"38cc399c-c456-40d8-9c11-24cd9a7f2a5d","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.003Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"62d37a5a-ee3d-4828-b943-8029acc3d6e0","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Error updating status","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","name":"raycluster-image-validation","error":"Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-image-validation\": the object has been modified; please apply your changes to the latest version and try again","RayCluster":{"apiVersion":"ray.io/v1","kind":"RayCluster","namespace":"test-ns-kppl8","name":"raycluster-image-validation"}} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d"} {"level":"error","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"69441f52-09b2-4ad2-b2a0-6d31b07a1b3d","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e"} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e"} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e"} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e","error":"head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"error","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.008Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e","name":"raycluster-image-validation","old status":{"desiredCPU":"0","desiredMemory":"0","desiredGPU":"0","desiredTPU":"0","head":{}},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:43Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:43.021Z","logger":"authentication-controller","msg":"Service account reconciled","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","name":"raycluster-image-validation-oauth-proxy-sa","operation":"created","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:02:43.080Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"62d37a5a-ee3d-4828-b943-8029acc3d6e0","name":"raycluster-image-validation-head"} {"level":"info","ts":"2026-09-11T13:02:43.080Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e"} {"level":"error","ts":"2026-09-11T13:02:43.080Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ae9e94c6-ea70-481a-9c65-9a7b9253565e","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.080Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb"} {"level":"info","ts":"2026-09-11T13:02:43.080Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb"} {"level":"info","ts":"2026-09-11T13:02:43.080Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb"} {"level":"info","ts":"2026-09-11T13:02:43.080Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:43.081Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb","error":"head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"error","ts":"2026-09-11T13:02:43.081Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.081Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"info","ts":"2026-09-11T13:02:43.081Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.081Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.081Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.081Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb"} {"level":"error","ts":"2026-09-11T13:02:43.081Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"8a60a4fc-281e-435a-8db8-cebcd69804fb","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.081Z","logger":"authentication-controller","msg":"ReferenceGrant reconciled (shared)","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","name":"kuberay-gateway-access","namespace":"test-ns-kppl8","operation":"created"} {"level":"info","ts":"2026-09-11T13:02:43.081Z","msg":"Self-signed issuer created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.081Z","msg":"Creating CA certificate for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.100Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f"} {"level":"info","ts":"2026-09-11T13:02:43.100Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f"} {"level":"info","ts":"2026-09-11T13:02:43.100Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f"} {"level":"info","ts":"2026-09-11T13:02:43.101Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:43.101Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f","error":"head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"error","ts":"2026-09-11T13:02:43.101Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.101Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"info","ts":"2026-09-11T13:02:43.101Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.101Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.101Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.101Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f"} {"level":"error","ts":"2026-09-11T13:02:43.101Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"231e4007-7b74-44ee-a9ab-094a567d7b7f","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.102Z","logger":"authentication-controller","msg":"HTTPRoute reconciled","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","name":"test-ns-kppl8-raycluster-image-validation","operation":"created"} {"level":"info","ts":"2026-09-11T13:02:43.116Z","msg":"CA Certificate created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","rayCluster":"raycluster-image-validation","secret":"ca-secret"} {"level":"info","ts":"2026-09-11T13:02:43.116Z","msg":"Creating CA issuer for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.116Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"62d37a5a-ee3d-4828-b943-8029acc3d6e0","name":"raycluster-image-validation-workers"} {"level":"info","ts":"2026-09-11T13:02:43.116Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"62d37a5a-ee3d-4828-b943-8029acc3d6e0","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.116Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"7a93e287-cc9d-4794-93f0-69c5f5f0c672","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.117Z","logger":"authentication-controller","msg":"OIDC ConfigMap reconciled","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","name":"kube-rbac-proxy-config-raycluster-image-validation","operation":"created"} {"level":"error","ts":"2026-09-11T13:02:43.135Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","error":"authentication resources created but failed to update AuthenticationReady condition: Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-image-validation\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.135Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8"} {"level":"error","ts":"2026-09-11T13:02:43.135Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"73d04e85-9d9f-45d8-bb48-e177ff7060b8","error":"authentication resources created but failed to update AuthenticationReady condition: Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-image-validation\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.135Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c3451526-ea1c-48bf-9ffb-774da106da48","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:02:43.135Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c3451526-ea1c-48bf-9ffb-774da106da48","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.135Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c3451526-ea1c-48bf-9ffb-774da106da48","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.180Z","msg":"CA issuer created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.180Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"7a93e287-cc9d-4794-93f0-69c5f5f0c672","name":"raycluster-image-validation-head"} {"level":"info","ts":"2026-09-11T13:02:43.181Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d"} {"level":"info","ts":"2026-09-11T13:02:43.181Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d"} {"level":"info","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d"} {"level":"info","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d","error":"head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"error","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"info","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d"} {"level":"error","ts":"2026-09-11T13:02:43.182Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d5fe6bb2-51b4-4f7a-b1ca-0e71f19b5e5d","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.202Z","logger":"authentication-controller","msg":"Successfully reconciled authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c3451526-ea1c-48bf-9ffb-774da106da48","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.202Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e86cf5ea-32a7-4ac6-a2c9-31084faed308","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:02:43.202Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e86cf5ea-32a7-4ac6-a2c9-31084faed308","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.202Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e86cf5ea-32a7-4ac6-a2c9-31084faed308","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.214Z","msg":"Ray head certificate created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","rayCluster":"raycluster-image-validation","podIPs":[]} {"level":"info","ts":"2026-09-11T13:02:43.218Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"7a93e287-cc9d-4794-93f0-69c5f5f0c672","name":"raycluster-image-validation-workers"} {"level":"info","ts":"2026-09-11T13:02:43.218Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"7a93e287-cc9d-4794-93f0-69c5f5f0c672","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.218Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f35ba246-5330-49f3-9324-9377dd795c71","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.221Z","logger":"authentication-controller","msg":"Successfully reconciled authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e86cf5ea-32a7-4ac6-a2c9-31084faed308","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.249Z","msg":"Ray worker certificate created successfully","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","rayCluster":"raycluster-image-validation","podIPs":[]} {"level":"error","ts":"2026-09-11T13:02:43.249Z","msg":"Failed to check certificate readiness","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","error":"Certificate.cert-manager.io \"ray-worker-cert-raycluster-image-validation\" not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:220\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.249Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09"} {"level":"error","ts":"2026-09-11T13:02:43.249Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f8b428fa-d4da-4d5d-9116-80c288378a09","error":"Certificate.cert-manager.io \"ray-worker-cert-raycluster-image-validation\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.249Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"3f3edf0f-853e-4a00-a5cb-8a62d8a61d90","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.249Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"3f3edf0f-853e-4a00-a5cb-8a62d8a61d90"} {"level":"info","ts":"2026-09-11T13:02:43.251Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f35ba246-5330-49f3-9324-9377dd795c71","name":"raycluster-image-validation-head"} {"level":"info","ts":"2026-09-11T13:02:43.280Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"93fbf94c-c5b1-4f21-96b3-fdf55a4aa8fb","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.280Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"93fbf94c-c5b1-4f21-96b3-fdf55a4aa8fb"} {"level":"info","ts":"2026-09-11T13:02:43.295Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f35ba246-5330-49f3-9324-9377dd795c71","name":"raycluster-image-validation-workers"} {"level":"info","ts":"2026-09-11T13:02:43.295Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f35ba246-5330-49f3-9324-9377dd795c71","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec"} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec"} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec"} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec","error":"head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"error","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec"} {"level":"error","ts":"2026-09-11T13:02:43.342Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"b35642dd-daa3-427f-a97c-a7424bc7e4ec","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a"} {"level":"info","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a"} {"level":"info","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a"} {"level":"info","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a","error":"head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"error","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"info","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.663Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:43.664Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a"} {"level":"error","ts":"2026-09-11T13:02:43.664Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"a12772ec-8c27-420a-a6f0-2b6e9ab15b8a","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:44.304Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7"} {"level":"info","ts":"2026-09-11T13:02:44.304Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7"} {"level":"info","ts":"2026-09-11T13:02:44.304Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7"} {"level":"info","ts":"2026-09-11T13:02:44.304Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:44.305Z","logger":"controllers.RayCluster","msg":"mTLS secrets not ready yet, requeuing","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7","error":"head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"error","ts":"2026-09-11T13:02:44.305Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:44.305Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found"} {"level":"info","ts":"2026-09-11T13:02:44.305Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:44.305Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:44.305Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:44.305Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7"} {"level":"error","ts":"2026-09-11T13:02:44.305Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"740ea715-9f6f-459f-a336-f6c4558d5cd7","error":"mTLS secrets not ready: head certificate secret ray-head-secret-raycluster-image-validation not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:02:45.585Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","enabled":true} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"Created head Pod for RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","name":"raycluster-image-validation-head-hsbkp"} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc"} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","nodeType":"worker","rayStartParams":{"address":"raycluster-image-validation-head-svc.test-ns-kppl8.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","rayStartCmd":"ray start --address=raycluster-image-validation-head-svc.test-ns-kppl8.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-image-validation-head-svc.test-ns-kppl8.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:45.713Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","enabled":true} {"level":"info","ts":"2026-09-11T13:02:45.801Z","logger":"controllers.RayCluster","msg":"Created worker Pod for RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","name":"raycluster-image-validation-small-group-worker-k44rt"} {"level":"info","ts":"2026-09-11T13:02:45.802Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:45.802Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5b1bb083-64c6-4c1e-89e0-37a6723cf0bc","name":"raycluster-image-validation","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:43Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:45Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podName":"raycluster-image-validation-head-hsbkp","serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939"} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939"} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939"} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:45.824Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f9a9fac7-e4f6-41cd-a4f5-307e7029b42f","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:45.824Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f9a9fac7-e4f6-41cd-a4f5-307e7029b42f","rayCluster":"raycluster-image-validation","podIPs":[]} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"8a7d7b38-840b-40d6-b3c2-843bbf7b7758","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:45.824Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:45.825Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:45.825Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","name":"raycluster-image-validation","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:43Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:45Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podName":"raycluster-image-validation-head-hsbkp","serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:45.843Z","logger":"controllers.RayCluster","msg":"Error updating status","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","name":"raycluster-image-validation","error":"Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-image-validation\": the object has been modified; please apply your changes to the latest version and try again","RayCluster":{"apiVersion":"ray.io/v1","kind":"RayCluster","namespace":"test-ns-kppl8","name":"raycluster-image-validation"}} {"level":"info","ts":"2026-09-11T13:02:45.843Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939"} {"level":"error","ts":"2026-09-11T13:02:45.843Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"819c9b60-b8bc-4a23-83e9-a17572309939","error":"Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-image-validation\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:45.843Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52"} {"level":"info","ts":"2026-09-11T13:02:45.843Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52"} {"level":"info","ts":"2026-09-11T13:02:45.843Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52"} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:45.844Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"46489409-be75-4e7d-b0f0-aba535352a52","seconds":300} {"level":"info","ts":"2026-09-11T13:02:45.849Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754"} {"level":"info","ts":"2026-09-11T13:02:45.849Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754"} {"level":"info","ts":"2026-09-11T13:02:45.849Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754"} {"level":"info","ts":"2026-09-11T13:02:45.849Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:45.850Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"24f7ce8e-8510-4154-a247-5c0325734754","seconds":300} {"level":"info","ts":"2026-09-11T13:02:45.858Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"8a7d7b38-840b-40d6-b3c2-843bbf7b7758","name":"raycluster-image-validation-head"} {"level":"info","ts":"2026-09-11T13:02:45.859Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7"} {"level":"info","ts":"2026-09-11T13:02:45.859Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7"} {"level":"info","ts":"2026-09-11T13:02:45.859Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7"} {"level":"info","ts":"2026-09-11T13:02:45.859Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:45.860Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:45.880Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:45.880Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"bda0239e-26c7-4cf2-af34-6d7b8e15d4b7","seconds":300} {"level":"info","ts":"2026-09-11T13:02:45.898Z","logger":"networkpolicy-controller","msg":"Successfully updated NetworkPolicy","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"8a7d7b38-840b-40d6-b3c2-843bbf7b7758","name":"raycluster-image-validation-workers"} {"level":"info","ts":"2026-09-11T13:02:45.898Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"8a7d7b38-840b-40d6-b3c2-843bbf7b7758","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:46.364Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"75938781-38ca-4e10-b617-b329a7d58ccd","seconds":300} {"level":"info","ts":"2026-09-11T13:02:46.441Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9"} {"level":"info","ts":"2026-09-11T13:02:46.441Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9"} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9"} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:46.442Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"96e93520-ba8b-4914-9f92-ad9d4fcf92f9","seconds":300} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01"} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01"} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01"} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:46.678Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:46.679Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:46.679Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:46.679Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b884aa1-2a7d-470b-a26f-a112fe4c1e01","name":"raycluster-image-validation","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:45Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podName":"raycluster-image-validation-head-hsbkp","serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:46Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.133.0.64","serviceIP":"10.133.0.64","podName":"raycluster-image-validation-head-hsbkp","serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:46.700Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"9dd9bf65-b756-43b1-ac4d-9fcfb3ca0bb5","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:46.700Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e0c19dba-cf79-4faa-b5e1-5dd6fcbac029","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:46.700Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"9dd9bf65-b756-43b1-ac4d-9fcfb3ca0bb5","pod":"raycluster-image-validation-head-hsbkp","ip":"10.133.0.64"} {"level":"info","ts":"2026-09-11T13:02:46.727Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e0c19dba-cf79-4faa-b5e1-5dd6fcbac029","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:02:46.736Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"9dd9bf65-b756-43b1-ac4d-9fcfb3ca0bb5","certificate":"ray-head-cert-raycluster-image-validation","dnsNames":["raycluster-image-validation-head-svc","localhost","raycluster-image-validation-head-svc.test-ns-kppl8.svc","raycluster-image-validation-head-svc.test-ns-kppl8.svc.cluster.local"],"ipAddresses":["10.133.0.64","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:02:46.771Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"9dd9bf65-b756-43b1-ac4d-9fcfb3ca0bb5","certificate":"ray-worker-cert-raycluster-image-validation","dnsNames":["raycluster-image-validation-worker-svc","localhost","raycluster-image-validation-small-group","raycluster-image-validation-worker-svc.test-ns-kppl8.svc","raycluster-image-validation-worker-svc.test-ns-kppl8.svc.cluster.local","*.raycluster-image-validation-worker-svc.test-ns-kppl8.svc","*.raycluster-image-validation-worker-svc.test-ns-kppl8.svc.cluster.local","*-worker-*.test-ns-kppl8.svc","*-worker-*.test-ns-kppl8.svc.cluster.local"],"ipAddresses":["10.133.0.64","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:02:46.771Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"9dd9bf65-b756-43b1-ac4d-9fcfb3ca0bb5","rayCluster":"raycluster-image-validation","podIPs":["10.133.0.64"]} {"level":"info","ts":"2026-09-11T13:02:46.999Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36"} {"level":"info","ts":"2026-09-11T13:02:46.999Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36"} {"level":"info","ts":"2026-09-11T13:02:46.999Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36"} {"level":"info","ts":"2026-09-11T13:02:46.999Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:47.000Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"1b3bb7a9-6956-4cad-9117-abf4b0252b36","seconds":300} {"level":"info","ts":"2026-09-11T13:02:47.753Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f"} {"level":"info","ts":"2026-09-11T13:02:47.753Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f"} {"level":"info","ts":"2026-09-11T13:02:47.753Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f"} {"level":"info","ts":"2026-09-11T13:02:47.753Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:47.753Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"ac31635a-a9f1-4387-a4c9-ac0f34f6462f","seconds":300} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33"} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33"} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33"} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:47.754Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.755Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:47.755Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"cbd90b61-037e-43e9-b913-276e0d5e9e33","seconds":300} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819"} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819"} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819"} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.781Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:47.782Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:47.782Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.782Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:47.782Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"72b4f477-b05f-494b-b8c8-3411d8965819","seconds":300} {"level":"info","ts":"2026-09-11T13:02:47.797Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4"} {"level":"info","ts":"2026-09-11T13:02:47.797Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4"} {"level":"info","ts":"2026-09-11T13:02:47.797Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4"} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-auth-token-small-group-worker-stx4v. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:47.798Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8bd2250f-4ea6-4b82-8e9d-173ddadacea4","seconds":300} {"level":"info","ts":"2026-09-11T13:02:48.501Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a"} {"level":"info","ts":"2026-09-11T13:02:48.501Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a"} {"level":"info","ts":"2026-09-11T13:02:48.501Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a"} {"level":"info","ts":"2026-09-11T13:02:48.501Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:48.502Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:48.502Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:48.502Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:48.502Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a","worker Pod":"raycluster-auth-token-small-group-worker-stx4v","shouldDelete":true,"reason":"The worker Pod raycluster-auth-token-small-group-worker-stx4v status is Failed which is a terminal state. KubeRay will delete the Pod and create new Pods in the next reconciliation if necessary."} {"level":"error","ts":"2026-09-11T13:02:48.516Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"delete 1 unhealthy worker Pods","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:48.516Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a","error":"delete 1 unhealthy worker Pods"} {"level":"info","ts":"2026-09-11T13:02:48.516Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a","name":"raycluster-auth-token","old status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:30Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"readyWorkerReplicas":1,"availableWorkerReplicas":1,"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:48Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:48.537Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a"} {"level":"error","ts":"2026-09-11T13:02:48.537Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"f71f7a99-301e-4b5b-b1d9-0ea30941db4a","error":"delete 1 unhealthy worker Pods","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:48.538Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"8362c396-a864-4e92-9b40-6c5300014340","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:48.538Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"6564838a-e7c3-417c-ac36-4d25e4719a52","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:48.538Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"6564838a-e7c3-417c-ac36-4d25e4719a52","pod":"raycluster-auth-token-head-zm8sh","ip":"10.134.0.43"} {"level":"info","ts":"2026-09-11T13:02:48.538Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"6564838a-e7c3-417c-ac36-4d25e4719a52","pod":"raycluster-auth-token-small-group-worker-stx4v","ip":"10.133.0.63"} {"level":"info","ts":"2026-09-11T13:02:48.538Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"6564838a-e7c3-417c-ac36-4d25e4719a52","rayCluster":"raycluster-auth-token","podIPs":["10.134.0.43","10.133.0.63"]} {"level":"info","ts":"2026-09-11T13:02:48.542Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e"} {"level":"info","ts":"2026-09-11T13:02:48.542Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e"} {"level":"info","ts":"2026-09-11T13:02:48.542Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e"} {"level":"info","ts":"2026-09-11T13:02:48.543Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:48.543Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:48.543Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:48.543Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e","worker group":"small-group","Expectation":"NotSatisfiedGroupExpectations, reconcile the group later"} {"level":"info","ts":"2026-09-11T13:02:48.543Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Failed"} {"level":"info","ts":"2026-09-11T13:02:48.543Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:48.543Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"d56c69b3-c070-47af-b387-ff7a7e28ac9e","seconds":300} {"level":"info","ts":"2026-09-11T13:02:48.564Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"8362c396-a864-4e92-9b40-6c5300014340","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:48.700Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97"} {"level":"info","ts":"2026-09-11T13:02:48.700Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97"} {"level":"info","ts":"2026-09-11T13:02:48.700Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97"} {"level":"info","ts":"2026-09-11T13:02:48.700Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:48.700Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:48.700Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:48.700Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97","worker group":"small-group","Expectation":"NotSatisfiedGroupExpectations, reconcile the group later"} {"level":"info","ts":"2026-09-11T13:02:48.700Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97","podName":"raycluster-auth-token-small-group-worker-stx4v","pod Status.Phase":"Failed"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"b27d43c4-326d-424b-9114-820c23b8ef97","seconds":300} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:48.701Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"59850c66-da7b-4323-a714-238906365904","seconds":300} {"level":"info","ts":"2026-09-11T13:02:48.702Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1"} {"level":"info","ts":"2026-09-11T13:02:48.702Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1"} {"level":"info","ts":"2026-09-11T13:02:48.702Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1"} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1"} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:48.703Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","enabled":true} {"level":"error","ts":"2026-09-11T13:02:48.717Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:48.717Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:48.717Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","name":"raycluster-auth-token","old status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:48Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:48Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:02:48Z","reason":"FailedCreateWorkerPod","message":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:48.741Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1"} {"level":"error","ts":"2026-09-11T13:02:48.741Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"bc12f244-0789-4e83-b251-657a162101e1","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:48.741Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"e23d5734-a5d1-476e-a6e1-f96967830305","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:48.741Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"b157b7e3-2059-4901-9db2-21953b3eeae2","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:48.741Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"e23d5734-a5d1-476e-a6e1-f96967830305","pod":"raycluster-auth-token-head-zm8sh","ip":"10.134.0.43"} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30"} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30"} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30"} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30"} {"level":"info","ts":"2026-09-11T13:02:48.780Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:48.781Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:48.781Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:48.781Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","enabled":true} {"level":"info","ts":"2026-09-11T13:02:48.781Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"e23d5734-a5d1-476e-a6e1-f96967830305","certificate":"ray-head-cert-raycluster-auth-token","dnsNames":["raycluster-auth-token-head-svc","localhost","raycluster-auth-token-head-svc.test-ns-r7tdx.svc","raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local"],"ipAddresses":["10.134.0.43","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:02:48.793Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"b157b7e3-2059-4901-9db2-21953b3eeae2","cluster":"raycluster-auth-token"} {"level":"error","ts":"2026-09-11T13:02:48.795Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:48.795Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:48.795Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30"} {"level":"error","ts":"2026-09-11T13:02:48.795Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"7545b711-3e49-46f8-aaf0-c09e1cc3aa30","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca"} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca"} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca"} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca"} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:48.806Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","enabled":true} {"level":"info","ts":"2026-09-11T13:02:48.814Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"e23d5734-a5d1-476e-a6e1-f96967830305","certificate":"ray-worker-cert-raycluster-auth-token","dnsNames":["raycluster-auth-token-worker-svc","localhost","raycluster-auth-token-small-group","raycluster-auth-token-worker-svc.test-ns-r7tdx.svc","raycluster-auth-token-worker-svc.test-ns-r7tdx.svc.cluster.local","*.raycluster-auth-token-worker-svc.test-ns-r7tdx.svc","*.raycluster-auth-token-worker-svc.test-ns-r7tdx.svc.cluster.local","*-worker-*.test-ns-r7tdx.svc","*-worker-*.test-ns-r7tdx.svc.cluster.local"],"ipAddresses":["10.134.0.43","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:02:48.814Z","msg":"All mTLS certificates are ready","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"e23d5734-a5d1-476e-a6e1-f96967830305","rayCluster":"raycluster-auth-token","podIPs":["10.134.0.43"]} {"level":"error","ts":"2026-09-11T13:02:48.820Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:48.820Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:48.820Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca"} {"level":"error","ts":"2026-09-11T13:02:48.820Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"3de093a5-faf7-49eb-bb39-70c9364111ca","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:48.841Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214"} {"level":"info","ts":"2026-09-11T13:02:48.841Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214"} {"level":"info","ts":"2026-09-11T13:02:48.841Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214"} {"level":"info","ts":"2026-09-11T13:02:48.841Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214"} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:48.842Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","enabled":true} {"level":"error","ts":"2026-09-11T13:02:48.886Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:48.886Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:48.980Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214"} {"level":"error","ts":"2026-09-11T13:02:48.980Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"86e57365-853a-4fde-8028-fc85570f0214","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f"} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f"} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f"} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f"} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:49.021Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","enabled":true} {"level":"error","ts":"2026-09-11T13:02:49.035Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.035Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:49.036Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f"} {"level":"error","ts":"2026-09-11T13:02:49.036Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6be8f1ad-90d4-4556-b230-11e6e12d3a9f","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393"} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393"} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393"} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:49.116Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:49.117Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:49.117Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393"} {"level":"info","ts":"2026-09-11T13:02:49.117Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:49.117Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:49.117Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:49.117Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","enabled":true} {"level":"error","ts":"2026-09-11T13:02:49.130Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.130Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:49.131Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393"} {"level":"error","ts":"2026-09-11T13:02:49.131Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"6c2c79ce-a049-4ffb-8ec6-cbcff3db5393","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4"} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4"} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4"} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-auth-token-head-zm8sh. The Pod status is Running, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4"} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:49.292Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","enabled":true} {"level":"error","ts":"2026-09-11T13:02:49.306Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.306Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:49.307Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4"} {"level":"error","ts":"2026-09-11T13:02:49.307Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"9a00b7c2-52f0-4d03-9c12-c78f0f9a3cd4","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.546Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb"} {"level":"info","ts":"2026-09-11T13:02:49.546Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb"} {"level":"info","ts":"2026-09-11T13:02:49.546Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb"} {"level":"info","ts":"2026-09-11T13:02:49.546Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:49.547Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb","Found 1 head Pod":"raycluster-auth-token-head-zm8sh","Pod status":"Failed","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2026-09-11 13:01:05 +0000 UTC,FinishedAt:2026-09-11 13:02:49 +0000 UTC,ContainerID:cri-o://c9ce54b02c00b4edc4a6df75214a6f25f83aaffb6459749ed562deac28eeddca,}"} {"level":"info","ts":"2026-09-11T13:02:49.547Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb","head Pod":"raycluster-auth-token-head-zm8sh","shouldDelete":true,"reason":"The head Pod raycluster-auth-token-head-zm8sh status is Failed which is a terminal state. KubeRay will delete the Pod and create new Pods in the next reconciliation if necessary."} {"level":"error","ts":"2026-09-11T13:02:49.560Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"The head Pod raycluster-auth-token-head-zm8sh status is Failed which is a terminal state. KubeRay will delete the Pod and create new Pods in the next reconciliation if necessary.","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.561Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb","error":"The head Pod raycluster-auth-token-head-zm8sh status is Failed which is a terminal state. KubeRay will delete the Pod and create new Pods in the next reconciliation if necessary."} {"level":"info","ts":"2026-09-11T13:02:49.561Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb","name":"raycluster-auth-token","old status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:48Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"True","lastTransitionTime":"2026-09-11T13:01:19Z","reason":"HeadPodRunningAndReady","message":""},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:02:48Z","reason":"FailedCreateWorkerPod","message":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:49Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:49Z","reason":"PodFailed","message":""},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:02:48Z","reason":"FailedCreateWorkerPod","message":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:49.583Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb"} {"level":"error","ts":"2026-09-11T13:02:49.583Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"e106495c-97ae-45ef-8298-302aa8e8fcbb","error":"The head Pod raycluster-auth-token-head-zm8sh status is Failed which is a terminal state. KubeRay will delete the Pod and create new Pods in the next reconciliation if necessary.","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.583Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0959540a-f9f0-4de5-a413-d1eab78e0a4f","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:49.583Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"e0e866e5-3da5-4a79-8298-916f13fa5a93","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:49.583Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0959540a-f9f0-4de5-a413-d1eab78e0a4f","pod":"raycluster-auth-token-head-zm8sh","ip":"10.134.0.43"} {"level":"info","ts":"2026-09-11T13:02:49.584Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0959540a-f9f0-4de5-a413-d1eab78e0a4f"} {"level":"info","ts":"2026-09-11T13:02:49.610Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"e0e866e5-3da5-4a79-8298-916f13fa5a93","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618"} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618"} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618"} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","Expectation":"NotSatisfiedHeadExpectations, reconcile head later"} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:49.627Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:49.628Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618"} {"level":"info","ts":"2026-09-11T13:02:49.628Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:49.628Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:49.628Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:49.628Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","enabled":true} {"level":"error","ts":"2026-09-11T13:02:49.641Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:49.642Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:49.642Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618"} {"level":"error","ts":"2026-09-11T13:02:49.642Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"604314cf-b394-4c51-a6d5-c43734fe4618","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e"} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e"} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e"} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","Expectation":"NotSatisfiedHeadExpectations, reconcile head later"} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","workerReplicas":1,"NumOfHosts":1,"runningPods":0,"diff":1} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","group":"small-group","indices":{}} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","Number workers to add":1,"Worker group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","creating worker for group":"small-group","index":0,"total":1,"replicaIndex":0} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for worker pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e"} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","nodeType":"worker","rayStartParams":{"address":"raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379","block":"true","dashboard-agent-listen-port":"52365","metrics-export-port":"8080","num-cpus":"1"},"Ray container resource":{"limits":{"cpu":"1","memory":"3G"},"requests":{"cpu":"500m","memory":"1G"}}} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","rayStartCmd":"ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","rayNodeType":"worker","generatedCmd":"ulimit -n 65536; ray start --address=raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local:6379 --block --dashboard-agent-listen-port=52365 --memory=3000000000 --metrics-export-port=8080 --num-cpus=1 "} {"level":"info","ts":"2026-09-11T13:02:50.040Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","enabled":true} {"level":"error","ts":"2026-09-11T13:02:50.054Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:50.054Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:50.055Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:50.055Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:50.055Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:50.055Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","name":"raycluster-auth-token","old status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:49Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:49Z","reason":"PodFailed","message":""},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:02:48Z","reason":"FailedCreateWorkerPod","message":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:50Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:49Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:02:48Z","reason":"FailedCreateWorkerPod","message":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e"} {"level":"error","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"8aeaa6b5-7aee-4a55-9f64-1f0494f9c24e","error":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","cluster":"raycluster-auth-token","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3c82c35e-963d-440d-87df-f6f57b4b6e4c","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:50.077Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"cd1f671b-9c34-415d-9249-22a33dd815c9","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:50.076Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","cluster":"raycluster-auth-token","authType":"IntegratedOAuth","serviceAccount":"raycluster-auth-token-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:02:50.077Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-auth-token","ray.io/group":"headgroup","ray.io/identifier":"raycluster-auth-token-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:02:50.077Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:02:50.077Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:02:50.077Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:02:50.077Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","enabled":true} {"level":"error","ts":"2026-09-11T13:02:50.090Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-auth-token-head-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:50.091Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","error":"FailedCreateHeadPod\npods \"raycluster-auth-token-head-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:50.091Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:50.091Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:50.091Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:50.091Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","name":"raycluster-auth-token","old status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:49Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.134.0.43","serviceIP":"10.134.0.43","podName":"raycluster-auth-token-head-zm8sh","serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:49Z","reason":"PodFailed","message":""},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:02:48Z","reason":"FailedCreateWorkerPod","message":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:50Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:49Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:02:48Z","reason":"FailedCreateHeadPod","message":"FailedCreateHeadPod\npods \"raycluster-auth-token-head-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:50.103Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"cd1f671b-9c34-415d-9249-22a33dd815c9","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:50.107Z","logger":"controllers.RayCluster","msg":"Error updating status","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","name":"raycluster-auth-token","error":"Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-auth-token\": the object has been modified; please apply your changes to the latest version and try again","RayCluster":{"apiVersion":"ray.io/v1","kind":"RayCluster","namespace":"test-ns-r7tdx","name":"raycluster-auth-token"}} {"level":"info","ts":"2026-09-11T13:02:50.107Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7"} {"level":"error","ts":"2026-09-11T13:02:50.107Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"77993d04-7872-45fd-b7e3-3892b5573dc7","error":"FailedCreateHeadPod\npods \"raycluster-auth-token-head-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:50.109Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3c82c35e-963d-440d-87df-f6f57b4b6e4c","certificate":"ray-head-cert-raycluster-auth-token","dnsNames":["raycluster-auth-token-head-svc","localhost","raycluster-auth-token-head-svc.test-ns-r7tdx.svc","raycluster-auth-token-head-svc.test-ns-r7tdx.svc.cluster.local"],"ipAddresses":["127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:02:50.142Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3c82c35e-963d-440d-87df-f6f57b4b6e4c","certificate":"ray-worker-cert-raycluster-auth-token","dnsNames":["raycluster-auth-token-worker-svc","localhost","raycluster-auth-token-small-group","raycluster-auth-token-worker-svc.test-ns-r7tdx.svc","raycluster-auth-token-worker-svc.test-ns-r7tdx.svc.cluster.local","*.raycluster-auth-token-worker-svc.test-ns-r7tdx.svc","*.raycluster-auth-token-worker-svc.test-ns-r7tdx.svc.cluster.local","*-worker-*.test-ns-r7tdx.svc","*-worker-*.test-ns-r7tdx.svc.cluster.local"],"ipAddresses":["127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:02:50.143Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3c82c35e-963d-440d-87df-f6f57b4b6e4c"} {"level":"info","ts":"2026-09-11T13:02:50.922Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373"} {"level":"info","ts":"2026-09-11T13:02:50.922Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373"} {"level":"info","ts":"2026-09-11T13:02:50.922Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373"} {"level":"info","ts":"2026-09-11T13:02:50.922Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","1 head service found":"raycluster-auth-token-head-svc"} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373"} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373"} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","cluster":"raycluster-auth-token","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","cluster":"raycluster-auth-token","authType":"IntegratedOAuth","serviceAccount":"raycluster-auth-token-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-auth-token","ray.io/group":"headgroup","ray.io/identifier":"raycluster-auth-token-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:02:50.923Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","enabled":true} {"level":"error","ts":"2026-09-11T13:02:50.937Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-auth-token-head-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:50.937Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","error":"FailedCreateHeadPod\npods \"raycluster-auth-token-head-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:50.937Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:50.937Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:50.937Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:50.937Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","name":"raycluster-auth-token","old status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:50Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:49Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:02:48Z","reason":"FailedCreateWorkerPod","message":"FailedCreateWorkerPod\npods \"raycluster-auth-token-small-group-worker-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"state":"ready","desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:50Z","stateTransitionTimes":{"ready":"2026-09-11T13:02:30Z"},"endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-auth-token-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:49Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"True","lastTransitionTime":"2026-09-11T13:02:30Z","reason":"AllPodRunningAndReadyFirstTime","message":"All Ray Pods are ready for the first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T12:58:22Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T12:58:23Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:02:48Z","reason":"FailedCreateHeadPod","message":"FailedCreateHeadPod\npods \"raycluster-auth-token-head-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:02:50.963Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373"} {"level":"error","ts":"2026-09-11T13:02:50.963Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"4a65049c-9220-40db-8583-25637838a373","error":"FailedCreateHeadPod\npods \"raycluster-auth-token-head-\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:50.963Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c9315680-fd53-446c-8023-3bc50e4ebb4c","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:50.963Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"5b09badb-c04c-4f0e-8761-4638a653b604","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:50.963Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c9315680-fd53-446c-8023-3bc50e4ebb4c"} {"level":"info","ts":"2026-09-11T13:02:50.989Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"5b09badb-c04c-4f0e-8761-4638a653b604","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:51.233Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"db423263-9163-4255-9ad0-03f6d3668630","seconds":300} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a"} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a"} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a"} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:51.245Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:51.246Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:51.246Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:51.246Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"d7645fcf-c9c1-40f8-9f45-e173c286250a","seconds":300} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6"} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6"} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6"} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:51.273Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"42f86dcf-ee00-4b04-a469-5cfc4dc355c6","seconds":300} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7"} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7"} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7"} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:02:51.276Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7c9192c5-6d60-4423-8378-ba17aadd4eb7","seconds":300} {"level":"info","ts":"2026-09-11T13:02:53.454Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3b827455-cbc4-44a5-894e-564b3bf25dc3","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T13:02:53.454Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3b827455-cbc4-44a5-894e-564b3bf25dc3","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.454Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3b827455-cbc4-44a5-894e-564b3bf25dc3","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.454Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868"} {"level":"info","ts":"2026-09-11T13:02:53.454Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868"} {"level":"info","ts":"2026-09-11T13:02:53.454Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868"} {"level":"error","ts":"2026-09-11T13:02:53.467Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcileHeadService-fm","error":"services \"raycluster-auth-token-head-svc\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.468Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868","error":"services \"raycluster-auth-token-head-svc\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated"} {"level":"info","ts":"2026-09-11T13:02:53.468Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:53.468Z","logger":"controllers.RayCluster","msg":"updateEndpoints: Unable to find a Service for this RayCluster. Not adding RayCluster status.endpoints","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868","serviceSelectors":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/identifier":"raycluster-auth-token-head","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:53.468Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868","filter labels":["test-ns-r7tdx",{"ray.io/cluster":"raycluster-auth-token","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:02:53.468Z","logger":"controllers.RayCluster","msg":"Got error when calculating new status","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868","error":"unable to find head service. cluster name raycluster-auth-token, filter labels [test-ns-r7tdx map[ray.io/cluster:raycluster-auth-token ray.io/identifier:raycluster-auth-token-head ray.io/node-type:head]]"} {"level":"info","ts":"2026-09-11T13:02:53.468Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868"} {"level":"error","ts":"2026-09-11T13:02:53.468Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"5cc0b05e-b5d4-4536-b9f7-8bba1a40f868","error":"services \"raycluster-auth-token-head-svc\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:02:53.489Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3b827455-cbc4-44a5-894e-564b3bf25dc3","error":"failed to ensure OIDC resources: failed to ensure ReferenceGrant: failed to create or update ReferenceGrant: referencegrants.gateway.networking.k8s.io \"kuberay-gateway-access\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.489Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3b827455-cbc4-44a5-894e-564b3bf25dc3"} {"level":"error","ts":"2026-09-11T13:02:53.489Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3b827455-cbc4-44a5-894e-564b3bf25dc3","error":"failed to ensure OIDC resources: failed to ensure ReferenceGrant: failed to create or update ReferenceGrant: referencegrants.gateway.networking.k8s.io \"kuberay-gateway-access\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.489Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0791c514-dfeb-40c7-b88f-f57b5d112e0a","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.489Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c36a21ff-6691-484d-b84b-b5165941dd8c","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.489Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0791c514-dfeb-40c7-b88f-f57b5d112e0a"} {"level":"info","ts":"2026-09-11T13:02:53.494Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"34dbdac4-52f7-4659-83e0-d5965b97892f","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T13:02:53.494Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"34dbdac4-52f7-4659-83e0-d5965b97892f","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.494Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"34dbdac4-52f7-4659-83e0-d5965b97892f","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.516Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"c36a21ff-6691-484d-b84b-b5165941dd8c","cluster":"raycluster-auth-token"} {"level":"error","ts":"2026-09-11T13:02:53.528Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"34dbdac4-52f7-4659-83e0-d5965b97892f","error":"failed to ensure OIDC resources: failed to ensure ReferenceGrant: failed to create or update ReferenceGrant: referencegrants.gateway.networking.k8s.io \"kuberay-gateway-access\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.528Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"34dbdac4-52f7-4659-83e0-d5965b97892f"} {"level":"error","ts":"2026-09-11T13:02:53.528Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"34dbdac4-52f7-4659-83e0-d5965b97892f","error":"failed to ensure OIDC resources: failed to ensure ReferenceGrant: failed to create or update ReferenceGrant: referencegrants.gateway.networking.k8s.io \"kuberay-gateway-access\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.538Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0419ecf5-12f3-4d00-9ad4-0a98815b0cad","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T13:02:53.538Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0419ecf5-12f3-4d00-9ad4-0a98815b0cad","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.538Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0419ecf5-12f3-4d00-9ad4-0a98815b0cad","cluster":"raycluster-auth-token"} {"level":"error","ts":"2026-09-11T13:02:53.571Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0419ecf5-12f3-4d00-9ad4-0a98815b0cad","error":"failed to ensure OIDC resources: failed to ensure ReferenceGrant: failed to create or update ReferenceGrant: referencegrants.gateway.networking.k8s.io \"kuberay-gateway-access\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.571Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0419ecf5-12f3-4d00-9ad4-0a98815b0cad"} {"level":"error","ts":"2026-09-11T13:02:53.571Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0419ecf5-12f3-4d00-9ad4-0a98815b0cad","error":"failed to ensure OIDC resources: failed to ensure ReferenceGrant: failed to create or update ReferenceGrant: referencegrants.gateway.networking.k8s.io \"kuberay-gateway-access\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.586Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0fad1dfa-239d-4fa3-93b8-d19672dc103b","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.586Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0fad1dfa-239d-4fa3-93b8-d19672dc103b"} {"level":"info","ts":"2026-09-11T13:02:53.592Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3082c5db-5a78-4d8c-9916-2dc5e2a86916","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T13:02:53.592Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3082c5db-5a78-4d8c-9916-2dc5e2a86916","mode":"IntegratedOAuth","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.592Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3082c5db-5a78-4d8c-9916-2dc5e2a86916","cluster":"raycluster-auth-token"} {"level":"error","ts":"2026-09-11T13:02:53.623Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3082c5db-5a78-4d8c-9916-2dc5e2a86916","error":"failed to ensure OIDC resources: failed to ensure ReferenceGrant: failed to create or update ReferenceGrant: referencegrants.gateway.networking.k8s.io \"kuberay-gateway-access\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.623Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3082c5db-5a78-4d8c-9916-2dc5e2a86916"} {"level":"error","ts":"2026-09-11T13:02:53.623Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"3082c5db-5a78-4d8c-9916-2dc5e2a86916","error":"failed to ensure OIDC resources: failed to ensure ReferenceGrant: failed to create or update ReferenceGrant: referencegrants.gateway.networking.k8s.io \"kuberay-gateway-access\" is forbidden: unable to create new content in namespace test-ns-r7tdx because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.658Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0649c943-351a-4817-a762-6bafa352504a","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T13:02:53.659Z","logger":"authentication-controller","msg":"RayCluster is being deleted, running finalizer","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0649c943-351a-4817-a762-6bafa352504a","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.659Z","logger":"authentication-controller","msg":"Running authentication resource cleanup (finalizer)","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0649c943-351a-4817-a762-6bafa352504a","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.659Z","logger":"controllers.RayCluster","msg":"RayCluster is being deleted, just ignore","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"reconcileID":"0d55aa5c-03b0-43d6-897a-57d4f6426459"} {"level":"info","ts":"2026-09-11T13:02:53.659Z","logger":"authentication-controller","msg":"ReferenceGrant cleanup check","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0649c943-351a-4817-a762-6bafa352504a","namespace":"test-ns-r7tdx","otherAuthClusters":0,"action":"will delete"} {"level":"info","ts":"2026-09-11T13:02:53.659Z","logger":"networkpolicy-controller","msg":"RayCluster is being deleted, NetworkPolicies will be garbage collected","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4c8098a5-9dc8-484a-9765-cb1b247894ff"} {"level":"info","ts":"2026-09-11T13:02:53.659Z","logger":"authentication-controller","msg":"ReferenceGrant already deleted","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0649c943-351a-4817-a762-6bafa352504a","name":"kuberay-gateway-access"} {"level":"info","ts":"2026-09-11T13:02:53.659Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"f2de51a3-e1fe-451b-915f-b542473b6f90","rayCluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.659Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"f2de51a3-e1fe-451b-915f-b542473b6f90"} {"level":"info","ts":"2026-09-11T13:02:53.675Z","logger":"authentication-controller","msg":"Deleted HTTPRoute","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0649c943-351a-4817-a762-6bafa352504a","name":"test-ns-r7tdx-raycluster-auth-token","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-09-11T13:02:53.690Z","logger":"authentication-controller","msg":"Deleted service account","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0649c943-351a-4817-a762-6bafa352504a","serviceAccount":"raycluster-auth-token-oauth-proxy-sa","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:02:53.707Z","logger":"raycluster-resource","msg":"default","name":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.707Z","logger":"raycluster-resource","msg":"enforcing secure trusted network on OpenShift","name":"raycluster-auth-token","namespace":"test-ns-r7tdx"} {"level":"info","ts":"2026-09-11T13:02:53.728Z","logger":"authentication-controller","msg":"Authentication cleanup complete, finalizer removed","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"0649c943-351a-4817-a762-6bafa352504a","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.728Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"88841289-bc29-4a9f-8d80-90d4c2aa53ad","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T13:02:53.728Z","logger":"authentication-controller","msg":"RayCluster is being deleted, running finalizer","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"88841289-bc29-4a9f-8d80-90d4c2aa53ad","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.728Z","logger":"authentication-controller","msg":"Running authentication resource cleanup (finalizer)","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"88841289-bc29-4a9f-8d80-90d4c2aa53ad","cluster":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.728Z","logger":"authentication-controller","msg":"ReferenceGrant cleanup check","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"88841289-bc29-4a9f-8d80-90d4c2aa53ad","namespace":"test-ns-r7tdx","otherAuthClusters":0,"action":"will delete"} {"level":"info","ts":"2026-09-11T13:02:53.728Z","logger":"authentication-controller","msg":"ReferenceGrant already deleted","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"88841289-bc29-4a9f-8d80-90d4c2aa53ad","name":"kuberay-gateway-access"} {"level":"info","ts":"2026-09-11T13:02:53.728Z","logger":"networkpolicy-controller","msg":"RayCluster not found, NetworkPolicies will be garbage collected","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"4795a88c-41b4-424a-8303-724da9422768"} {"level":"info","ts":"2026-09-11T13:02:53.728Z","msg":"Cleaning up MTLS resources for deleted RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"71c647d1-aacc-4e6f-86c6-0a14fad45fef","namespace":"test-ns-r7tdx","clusterName":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.733Z","logger":"networkpolicy-controller","msg":"RayCluster not found, NetworkPolicies will be garbage collected","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"20768764-6ef9-41d2-9ed4-b2fdce58afad"} {"level":"info","ts":"2026-09-11T13:02:53.734Z","logger":"networkpolicy-controller","msg":"RayCluster not found, NetworkPolicies will be garbage collected","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"33fe34a1-b13f-4ed3-a1e5-9d6718d88480"} {"level":"error","ts":"2026-09-11T13:02:53.742Z","msg":"Failed to delete CA issuer","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"71c647d1-aacc-4e6f-86c6-0a14fad45fef","issuer":"ray-ca-issuer-raycluster-auth-token","error":"issuers.cert-manager.io \"ray-ca-issuer-raycluster-auth-token\" not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).cleanupMTLSResources\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:247\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:144\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:02:53.742Z","logger":"authentication-controller","msg":"Failed to remove finalizer","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"88841289-bc29-4a9f-8d80-90d4c2aa53ad","error":"rayclusters.ray.io \"raycluster-auth-token\" not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).handleDeletion\n\t/workspace/controllers/ray/authentication_controller.go:196\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:137\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.742Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"88841289-bc29-4a9f-8d80-90d4c2aa53ad"} {"level":"error","ts":"2026-09-11T13:02:53.742Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"88841289-bc29-4a9f-8d80-90d4c2aa53ad","error":"rayclusters.ray.io \"raycluster-auth-token\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.742Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"5e39dccb-1071-4796-911f-25e3f5bd0cc5","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T13:02:53.742Z","logger":"authentication-controller","msg":"RayCluster not found, likely deleted - checking for orphaned resources","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"5e39dccb-1071-4796-911f-25e3f5bd0cc5"} {"level":"info","ts":"2026-09-11T13:02:53.742Z","logger":"authentication-controller","msg":"Checking for orphaned ReferenceGrant","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"5e39dccb-1071-4796-911f-25e3f5bd0cc5","namespace":"test-ns-r7tdx","authClusters":0} {"level":"info","ts":"2026-09-11T13:02:53.742Z","logger":"authentication-controller","msg":"No orphaned ReferenceGrant found","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"5e39dccb-1071-4796-911f-25e3f5bd0cc5","namespace":"test-ns-r7tdx"} {"level":"info","ts":"2026-09-11T13:02:53.748Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"eeb727fb-2f9e-45a7-9a7b-6b54f889d15c","namespacedName":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"}} {"level":"info","ts":"2026-09-11T13:02:53.748Z","logger":"authentication-controller","msg":"RayCluster not found, likely deleted - checking for orphaned resources","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"eeb727fb-2f9e-45a7-9a7b-6b54f889d15c"} {"level":"info","ts":"2026-09-11T13:02:53.748Z","logger":"authentication-controller","msg":"Checking for orphaned ReferenceGrant","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"eeb727fb-2f9e-45a7-9a7b-6b54f889d15c","namespace":"test-ns-r7tdx","authClusters":0} {"level":"info","ts":"2026-09-11T13:02:53.748Z","logger":"authentication-controller","msg":"No orphaned ReferenceGrant found","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"eeb727fb-2f9e-45a7-9a7b-6b54f889d15c","namespace":"test-ns-r7tdx"} {"level":"info","ts":"2026-09-11T13:02:53.838Z","msg":"Some cleanup operations failed, will retry","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"71c647d1-aacc-4e6f-86c6-0a14fad45fef","errorCount":1} {"level":"info","ts":"2026-09-11T13:02:53.838Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"71c647d1-aacc-4e6f-86c6-0a14fad45fef"} {"level":"error","ts":"2026-09-11T13:02:53.838Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"71c647d1-aacc-4e6f-86c6-0a14fad45fef","error":"issuers.cert-manager.io \"ray-ca-issuer-raycluster-auth-token\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:02:53.843Z","msg":"Cleaning up MTLS resources for deleted RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"e736199c-bf22-4fa6-b0d8-51f05a548e16","namespace":"test-ns-r7tdx","clusterName":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:02:53.843Z","msg":"Successfully cleaned up MTLS resources","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-auth-token","namespace":"test-ns-r7tdx"},"namespace":"test-ns-r7tdx","name":"raycluster-auth-token","reconcileID":"e736199c-bf22-4fa6-b0d8-51f05a548e16","namespace":"test-ns-r7tdx","clusterName":"raycluster-auth-token"} {"level":"info","ts":"2026-09-11T13:03:13.280Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4eedb9a3-8f57-4164-b73d-feaaf63a5c46","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:13.280Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4eedb9a3-8f57-4164-b73d-feaaf63a5c46","pod":"raycluster-image-validation-head-hsbkp","ip":"10.133.0.64"} {"level":"info","ts":"2026-09-11T13:03:13.280Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4eedb9a3-8f57-4164-b73d-feaaf63a5c46","pod":"raycluster-image-validation-small-group-worker-k44rt","ip":"10.134.0.44"} {"level":"info","ts":"2026-09-11T13:03:13.338Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4eedb9a3-8f57-4164-b73d-feaaf63a5c46","certificate":"ray-head-cert-raycluster-image-validation","dnsNames":["raycluster-image-validation-head-svc","localhost","raycluster-image-validation-head-svc.test-ns-kppl8.svc","raycluster-image-validation-head-svc.test-ns-kppl8.svc.cluster.local"],"ipAddresses":["10.133.0.64","10.134.0.44","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:03:13.392Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4eedb9a3-8f57-4164-b73d-feaaf63a5c46","certificate":"ray-worker-cert-raycluster-image-validation","dnsNames":["raycluster-image-validation-worker-svc","localhost","raycluster-image-validation-small-group","raycluster-image-validation-worker-svc.test-ns-kppl8.svc","raycluster-image-validation-worker-svc.test-ns-kppl8.svc.cluster.local","*.raycluster-image-validation-worker-svc.test-ns-kppl8.svc","*.raycluster-image-validation-worker-svc.test-ns-kppl8.svc.cluster.local","*-worker-*.test-ns-kppl8.svc","*-worker-*.test-ns-kppl8.svc.cluster.local"],"ipAddresses":["10.133.0.64","10.134.0.44","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:03:13.392Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4eedb9a3-8f57-4164-b73d-feaaf63a5c46"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","podName":"raycluster-image-validation-head-hsbkp","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:03:21.475Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2a692f77-4934-456b-9163-e17b4604cdb6","seconds":300} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb"} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb"} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb"} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Pending","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-image-validation-head-hsbkp. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"small-group","maxReplicas":1,"minReplicas":1,"replicas":1} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","worker Pod":"raycluster-image-validation-small-group-worker-k44rt","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-image-validation-small-group-worker-k44rt. The Pod status is Pending, and the Ray container terminated status is nil."} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","removing the pods in the scaleStrategy of":"small-group"} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","workerReplicas":1,"NumOfHosts":1,"runningPods":1,"diff":0} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"reconcilePods: found existing replica indices","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","group":"small-group","indices":{"0":true}} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","all workers already exist for group":"small-group"} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","podName":"raycluster-image-validation-small-group-worker-k44rt","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","podName":"raycluster-image-validation-small-group-worker-k44rt","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"Environment variable is not set, using default value of seconds","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","environmentVariable":"RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV","defaultValue":300} {"level":"info","ts":"2026-09-11T13:03:21.503Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"90c9c0e7-2b12-4205-99a5-768b8edd8ccb","seconds":300} {"level":"info","ts":"2026-09-11T13:03:21.936Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6"} {"level":"info","ts":"2026-09-11T13:03:21.936Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6"} {"level":"info","ts":"2026-09-11T13:03:21.936Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6"} {"level":"info","ts":"2026-09-11T13:03:21.936Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:21.936Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","Found 1 head Pod":"raycluster-image-validation-head-hsbkp","Pod status":"Failed","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"} {"level":"info","ts":"2026-09-11T13:03:21.936Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","head Pod":"raycluster-image-validation-head-hsbkp","shouldDelete":true,"reason":"The head Pod raycluster-image-validation-head-hsbkp status is Failed which is a terminal state. KubeRay will delete the Pod and create new Pods in the next reconciliation if necessary."} {"level":"error","ts":"2026-09-11T13:03:21.950Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedDeleteHeadPod\npods \"raycluster-image-validation-head-hsbkp\" not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:21.950Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","error":"FailedDeleteHeadPod\npods \"raycluster-image-validation-head-hsbkp\" not found"} {"level":"info","ts":"2026-09-11T13:03:21.951Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:21.951Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","podName":"raycluster-image-validation-small-group-worker-k44rt","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:21.951Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:21.951Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:21.951Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","name":"raycluster-image-validation","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:02:46Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"podIP":"10.133.0.64","serviceIP":"10.133.0.64","podName":"raycluster-image-validation-head-hsbkp","serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"PodInitializing","message":"containers with unready status: [ray-head kube-rbac-proxy]; kube-rbac-proxy: "},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:03:21Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:03:21Z","reason":"FailedDeleteHeadPod","message":"FailedDeleteHeadPod\npods \"raycluster-image-validation-head-hsbkp\" not found"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:03:21.972Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6"} {"level":"error","ts":"2026-09-11T13:03:21.972Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"7b1c54a5-f9fd-4299-a92f-a81f7a347ac6","error":"FailedDeleteHeadPod\npods \"raycluster-image-validation-head-hsbkp\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:21.972Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"0e44783c-8633-43db-ae65-84bcbe3c9431","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:21.972Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"59df8c76-c73a-4326-ab03-1468a513bf9f","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:21.972Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"59df8c76-c73a-4326-ab03-1468a513bf9f","pod":"raycluster-image-validation-small-group-worker-k44rt","ip":"10.134.0.44"} {"level":"info","ts":"2026-09-11T13:03:21.972Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150"} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150"} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150"} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150"} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150"} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:21.973Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","enabled":true} {"level":"error","ts":"2026-09-11T13:03:21.987Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:21.987Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:21.987Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:21.987Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","podName":"raycluster-image-validation-small-group-worker-k44rt","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:21.987Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:21.987Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:21.987Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","name":"raycluster-image-validation","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:03:21Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:03:21Z","reason":"FailedDeleteHeadPod","message":"FailedDeleteHeadPod\npods \"raycluster-image-validation-head-hsbkp\" not found"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:03:21Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:03:21Z","reason":"FailedCreateHeadPod","message":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:03:21.999Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"0e44783c-8633-43db-ae65-84bcbe3c9431","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.005Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"59df8c76-c73a-4326-ab03-1468a513bf9f","certificate":"ray-head-cert-raycluster-image-validation","dnsNames":["raycluster-image-validation-head-svc","localhost","raycluster-image-validation-head-svc.test-ns-kppl8.svc","raycluster-image-validation-head-svc.test-ns-kppl8.svc.cluster.local"],"ipAddresses":["10.134.0.44","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150"} {"level":"error","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"f3bfaece-3d13-4ddc-af41-659e0c5e0150","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","enabled":true} {"level":"info","ts":"2026-09-11T13:03:22.011Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"0189ab64-b665-467d-9844-61a2e51e9cc2","cluster":"raycluster-image-validation"} {"level":"error","ts":"2026-09-11T13:03:22.025Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.025Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:22.025Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.025Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","podName":"raycluster-image-validation-small-group-worker-k44rt","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:22.025Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.025Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.025Z","logger":"controllers.RayCluster","msg":"updateRayClusterStatus","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","name":"raycluster-image-validation","old status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:03:21Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:03:21Z","reason":"FailedDeleteHeadPod","message":"FailedDeleteHeadPod\npods \"raycluster-image-validation-head-hsbkp\" not found"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1},"new status":{"desiredCPU":"1","desiredMemory":"7G","desiredGPU":"0","desiredTPU":"0","lastUpdateTime":"2026-09-11T13:03:22Z","endpoints":{"client":"10001","dashboard":"8265","gcs-server":"6379","metrics":"8080","serve":"8000"},"head":{"serviceName":"raycluster-image-validation-head-svc"},"conditions":[{"type":"HeadPodReady","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"HeadPodNotFound","message":"Head Pod not found"},{"type":"RayClusterProvisioned","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterPodsProvisioning","message":"RayCluster Pods are being provisioned for first time"},{"type":"RayClusterSuspended","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspended","message":""},{"type":"RayClusterSuspending","status":"False","lastTransitionTime":"2026-09-11T13:02:43Z","reason":"RayClusterSuspending","message":""},{"type":"AuthenticationReady","status":"True","observedGeneration":1,"lastTransitionTime":"2026-09-11T13:02:43Z","reason":"AuthenticationResourcesCreated","message":"Authentication resources created successfully (mode: IntegratedOAuth)"},{"type":"ReplicaFailure","status":"True","lastTransitionTime":"2026-09-11T13:03:21Z","reason":"FailedCreateHeadPod","message":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"}],"desiredWorkerReplicas":1,"minWorkerReplicas":1,"maxWorkerReplicas":1,"observedGeneration":1}} {"level":"info","ts":"2026-09-11T13:03:22.038Z","msg":"Updated certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"59df8c76-c73a-4326-ab03-1468a513bf9f","certificate":"ray-worker-cert-raycluster-image-validation","dnsNames":["raycluster-image-validation-worker-svc","localhost","raycluster-image-validation-small-group","raycluster-image-validation-worker-svc.test-ns-kppl8.svc","raycluster-image-validation-worker-svc.test-ns-kppl8.svc.cluster.local","*.raycluster-image-validation-worker-svc.test-ns-kppl8.svc","*.raycluster-image-validation-worker-svc.test-ns-kppl8.svc.cluster.local","*-worker-*.test-ns-kppl8.svc","*-worker-*.test-ns-kppl8.svc.cluster.local"],"ipAddresses":["10.134.0.44","127.0.0.1"]} {"level":"info","ts":"2026-09-11T13:03:22.038Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"59df8c76-c73a-4326-ab03-1468a513bf9f"} {"level":"info","ts":"2026-09-11T13:03:22.038Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"84890ed8-5591-4f8a-a340-e712fa65d80a","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.038Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"84890ed8-5591-4f8a-a340-e712fa65d80a","pod":"raycluster-image-validation-small-group-worker-k44rt","ip":"10.134.0.44"} {"level":"info","ts":"2026-09-11T13:03:22.038Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"0189ab64-b665-467d-9844-61a2e51e9cc2","cluster":"raycluster-image-validation"} {"level":"error","ts":"2026-09-11T13:03:22.080Z","msg":"Failed to update certificate SANs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"84890ed8-5591-4f8a-a340-e712fa65d80a","error":"Operation cannot be fulfilled on certificates.cert-manager.io \"ray-worker-cert-raycluster-image-validation\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).updateCertificateWithPodIPs\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:465\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:212\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:03:22.080Z","msg":"Failed to update worker certificate with pod IPs","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"84890ed8-5591-4f8a-a340-e712fa65d80a","error":"Operation cannot be fulfilled on certificates.cert-manager.io \"ray-worker-cert-raycluster-image-validation\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:213\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.080Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"84890ed8-5591-4f8a-a340-e712fa65d80a"} {"level":"error","ts":"2026-09-11T13:03:22.080Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"84890ed8-5591-4f8a-a340-e712fa65d80a","error":"Operation cannot be fulfilled on certificates.cert-manager.io \"ray-worker-cert-raycluster-image-validation\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.080Z","logger":"controllers.RayCluster","msg":"Error updating status","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","name":"raycluster-image-validation","error":"Operation cannot be fulfilled on rayclusters.ray.io \"raycluster-image-validation\": the object has been modified; please apply your changes to the latest version and try again","RayCluster":{"apiVersion":"ray.io/v1","kind":"RayCluster","namespace":"test-ns-kppl8","name":"raycluster-image-validation"}} {"level":"info","ts":"2026-09-11T13:03:22.080Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5"} {"level":"error","ts":"2026-09-11T13:03:22.080Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"527dbba7-445e-461b-866e-2b7110a1bda5","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.080Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc"} {"level":"info","ts":"2026-09-11T13:03:22.080Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc"} {"level":"info","ts":"2026-09-11T13:03:22.080Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc"} {"level":"info","ts":"2026-09-11T13:03:22.080Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc"} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc"} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.081Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","enabled":true} {"level":"info","ts":"2026-09-11T13:03:22.086Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"cc140bdf-51aa-4f4b-89b6-ef54ee79c2b1","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.086Z","msg":"Found pod IP","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"cc140bdf-51aa-4f4b-89b6-ef54ee79c2b1","pod":"raycluster-image-validation-small-group-worker-k44rt","ip":"10.134.0.44"} {"level":"info","ts":"2026-09-11T13:03:22.086Z","msg":"One or more certificates are not ready, requeuing","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"cc140bdf-51aa-4f4b-89b6-ef54ee79c2b1"} {"level":"error","ts":"2026-09-11T13:03:22.095Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.095Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:22.095Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.095Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","podName":"raycluster-image-validation-small-group-worker-k44rt","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:22.095Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.095Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.095Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc"} {"level":"error","ts":"2026-09-11T13:03:22.095Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"119b9da8-8982-4932-8930-2b45224f5ddc","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:22.101Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:22.102Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.102Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.102Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","enabled":true} {"level":"error","ts":"2026-09-11T13:03:22.115Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.115Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:22.115Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.115Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","podName":"raycluster-image-validation-small-group-worker-k44rt","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:22.116Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.116Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.116Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4"} {"level":"error","ts":"2026-09-11T13:03:22.116Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"ad677902-6437-4177-a48f-60cde34481d4","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.197Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","enabled":true} {"level":"error","ts":"2026-09-11T13:03:22.211Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.211Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:22.211Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.211Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","podName":"raycluster-image-validation-small-group-worker-k44rt","pod Status.Phase":"Pending"} {"level":"info","ts":"2026-09-11T13:03:22.211Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.211Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.211Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0"} {"level":"error","ts":"2026-09-11T13:03:22.211Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"744aaef6-d5a3-4321-a3e7-7d5891a656e0","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.218Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","enabled":true} {"level":"error","ts":"2026-09-11T13:03:22.232Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.232Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:22.233Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.233Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","podName":"raycluster-image-validation-small-group-worker-k44rt","pod Status.Phase":"Failed"} {"level":"info","ts":"2026-09-11T13:03:22.233Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.233Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.233Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67"} {"level":"error","ts":"2026-09-11T13:03:22.233Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"5ba63d10-deb8-426c-887e-a5ae485faf67","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.235Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","enabled":true} {"level":"error","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3"} {"level":"error","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"97ca69c7-801d-42b3-85ab-f6f06fc125d3","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.249Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","enabled":true} {"level":"error","ts":"2026-09-11T13:03:22.263Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.263Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:22.263Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.263Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.263Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.263Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8"} {"level":"error","ts":"2026-09-11T13:03:22.263Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"94c805d0-0a70-4f48-9196-abaa8c708fb8","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.371Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad"} {"level":"info","ts":"2026-09-11T13:03:22.371Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad"} {"level":"info","ts":"2026-09-11T13:03:22.371Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad"} {"level":"info","ts":"2026-09-11T13:03:22.371Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad"} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad"} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:22.372Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","enabled":true} {"level":"error","ts":"2026-09-11T13:03:22.386Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:22.386Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:22.386Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.386Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.386Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:22.386Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad"} {"level":"error","ts":"2026-09-11T13:03:22.386Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"4bdbbaff-880c-46ae-a176-58a6f5b3e3ad","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","1 head service found":"raycluster-image-validation-head-svc"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"reconcilePods: Found 0 head Pods; creating a head Pod for the RayCluster.","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"AuthenticationReady condition is True and fresh, proceeding with pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","reason":"AuthenticationResourcesCreated","observedGeneration":1,"authMode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"mTLS is enabled, configuring mTLS for head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"Detected authentication mode for pod creation","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"Injecting OIDC proxy sidecar (kube-rbac-proxy)","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","cluster":"raycluster-image-validation","mode":"IntegratedOAuth"} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"Authentication sidecar injected successfully","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","cluster":"raycluster-image-validation","authType":"IntegratedOAuth","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","containerCount":2} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"head pod labels","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","labels":{"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","ray.io/cluster":"raycluster-image-validation","ray.io/group":"headgroup","ray.io/identifier":"raycluster-image-validation-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head"}} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","nodeType":"head","rayStartParams":{"block":"true","dashboard-agent-listen-port":"52365","dashboard-host":"0.0.0.0","metrics-export-port":"8080"},"Ray container resource":{"limits":{"cpu":"2","memory":"10G"},"requests":{"cpu":"500m","memory":"6G"}}} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"generateRayStartCommand","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","rayStartCmd":"ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"BuildPod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","rayNodeType":"head","generatedCmd":"ulimit -n 65536; ray start --head --block --dashboard-agent-listen-port=52365 --dashboard-host=0.0.0.0 --memory=10000000000 --metrics-export-port=8080 --num-cpus=2 "} {"level":"info","ts":"2026-09-11T13:03:24.946Z","logger":"controllers.RayCluster","msg":"Probes injection feature flag","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","enabled":true} {"level":"error","ts":"2026-09-11T13:03:24.960Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcilePods-fm","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:24.960Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:24.961Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:24.961Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:24.961Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:24.961Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5"} {"level":"error","ts":"2026-09-11T13:03:24.961Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2eaf1619-c8eb-4a61-a622-f1557dc5bcc5","error":"FailedCreateHeadPod\npods \"raycluster-image-validation-head-\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:28.304Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"aa779e73-a8e1-4ce0-a818-0785055359c2","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:28.331Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"aa779e73-a8e1-4ce0-a818-0785055359c2","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:28.331Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"1acbd6aa-d2a9-45ca-af82-af6aab63fb30","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:28.357Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"1acbd6aa-d2a9-45ca-af82-af6aab63fb30","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.033Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"277ec846-2639-4026-8c71-466dafcc76b7","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:03:29.033Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"277ec846-2639-4026-8c71-466dafcc76b7","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.033Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"277ec846-2639-4026-8c71-466dafcc76b7","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.033Z","logger":"controllers.RayCluster","msg":"Reconciling Ingress","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834"} {"level":"info","ts":"2026-09-11T13:03:29.033Z","logger":"controllers.RayCluster","msg":"EnableIngress is false, cleaning up any existing Routes/Ingresses","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834"} {"level":"info","ts":"2026-09-11T13:03:29.033Z","logger":"controllers.RayCluster","msg":"Reconciling Auth","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834"} {"level":"error","ts":"2026-09-11T13:03:29.047Z","logger":"controllers.RayCluster","msg":"Error reconcile resources","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834","function name":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).reconcileHeadService-fm","error":"services \"raycluster-image-validation-head-svc\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).rayClusterReconcile\n\t/workspace/controllers/ray/raycluster_controller.go:332\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterReconciler).Reconcile\n\t/workspace/controllers/ray/raycluster_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.047Z","logger":"controllers.RayCluster","msg":"Reconciliation error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834","error":"services \"raycluster-image-validation-head-svc\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated"} {"level":"info","ts":"2026-09-11T13:03:29.047Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:29.047Z","logger":"controllers.RayCluster","msg":"updateEndpoints: Unable to find a Service for this RayCluster. Not adding RayCluster status.endpoints","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834","serviceSelectors":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/identifier":"raycluster-image-validation-head","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:29.047Z","logger":"controllers.RayCluster","msg":"Found 0 head pod","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834","filter labels":["test-ns-kppl8",{"ray.io/cluster":"raycluster-image-validation","ray.io/node-type":"head"}]} {"level":"info","ts":"2026-09-11T13:03:29.047Z","logger":"controllers.RayCluster","msg":"Got error when calculating new status","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834","error":"unable to find head service. cluster name raycluster-image-validation, filter labels [test-ns-kppl8 map[ray.io/cluster:raycluster-image-validation ray.io/identifier:raycluster-image-validation-head ray.io/node-type:head]]"} {"level":"info","ts":"2026-09-11T13:03:29.047Z","logger":"controllers.RayCluster","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834"} {"level":"error","ts":"2026-09-11T13:03:29.047Z","logger":"controllers.RayCluster","msg":"Reconciler error","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"072444e5-c7ba-4fa5-b250-28a7757e7834","error":"services \"raycluster-image-validation-head-svc\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:03:29.069Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"277ec846-2639-4026-8c71-466dafcc76b7","error":"failed to ensure OIDC resources: failed to ensure ConfigMap: failed to create or update OIDC ConfigMap: configmaps \"kube-rbac-proxy-config-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.069Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"277ec846-2639-4026-8c71-466dafcc76b7"} {"level":"error","ts":"2026-09-11T13:03:29.069Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"277ec846-2639-4026-8c71-466dafcc76b7","error":"failed to ensure OIDC resources: failed to ensure ConfigMap: failed to create or update OIDC ConfigMap: configmaps \"kube-rbac-proxy-config-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.069Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"a4992be0-e54b-4636-b7b0-5c380381a5e1","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.069Z","logger":"networkpolicy-controller","msg":"Reconciling NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c6995fab-2fcf-4430-b4f0-04a6e615b2b7","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.069Z","msg":"Creating CA certificate for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"a4992be0-e54b-4636-b7b0-5c380381a5e1","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.074Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e0288aed-e39a-4800-8ef8-a94a216405fb","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:03:29.074Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e0288aed-e39a-4800-8ef8-a94a216405fb","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.074Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e0288aed-e39a-4800-8ef8-a94a216405fb","cluster":"raycluster-image-validation"} {"level":"error","ts":"2026-09-11T13:03:29.082Z","msg":"Failed to create CA Certificate","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"a4992be0-e54b-4636-b7b0-5c380381a5e1","error":"certificates.cert-manager.io \"ray-ca-certificate-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).createCACertificate\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:890\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).reconcileRootCA\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:483\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:165\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:03:29.082Z","msg":"Failed to reconcile RootCA","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"a4992be0-e54b-4636-b7b0-5c380381a5e1","error":"certificates.cert-manager.io \"ray-ca-certificate-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:166\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.082Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"a4992be0-e54b-4636-b7b0-5c380381a5e1"} {"level":"error","ts":"2026-09-11T13:03:29.082Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"a4992be0-e54b-4636-b7b0-5c380381a5e1","error":"certificates.cert-manager.io \"ray-ca-certificate-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.087Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c678d63f-a341-402d-905d-03a7f50c1e32","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.087Z","msg":"Creating CA certificate for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c678d63f-a341-402d-905d-03a7f50c1e32","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.095Z","logger":"networkpolicy-controller","msg":"Successfully reconciled NetworkPolicies for RayCluster","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c6995fab-2fcf-4430-b4f0-04a6e615b2b7","cluster":"raycluster-image-validation"} {"level":"error","ts":"2026-09-11T13:03:29.100Z","msg":"Failed to create CA Certificate","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c678d63f-a341-402d-905d-03a7f50c1e32","error":"certificates.cert-manager.io \"ray-ca-certificate-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).createCACertificate\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:890\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).reconcileRootCA\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:483\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:165\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:03:29.100Z","msg":"Failed to reconcile RootCA","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c678d63f-a341-402d-905d-03a7f50c1e32","error":"certificates.cert-manager.io \"ray-ca-certificate-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:166\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.100Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c678d63f-a341-402d-905d-03a7f50c1e32"} {"level":"error","ts":"2026-09-11T13:03:29.100Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c678d63f-a341-402d-905d-03a7f50c1e32","error":"certificates.cert-manager.io \"ray-ca-certificate-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:03:29.106Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e0288aed-e39a-4800-8ef8-a94a216405fb","error":"failed to ensure OIDC resources: failed to ensure ConfigMap: failed to create or update OIDC ConfigMap: configmaps \"kube-rbac-proxy-config-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.106Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e0288aed-e39a-4800-8ef8-a94a216405fb"} {"level":"error","ts":"2026-09-11T13:03:29.106Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"e0288aed-e39a-4800-8ef8-a94a216405fb","error":"failed to ensure OIDC resources: failed to ensure ConfigMap: failed to create or update OIDC ConfigMap: configmaps \"kube-rbac-proxy-config-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.111Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f06dacc6-ec22-4827-b122-124fdaa2d27c","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.111Z","msg":"Creating CA certificate for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f06dacc6-ec22-4827-b122-124fdaa2d27c","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.117Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"12570f3b-d442-4092-9967-ce649b1c4f8b","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:03:29.117Z","logger":"authentication-controller","msg":"Detected authentication mode","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"12570f3b-d442-4092-9967-ce649b1c4f8b","mode":"IntegratedOAuth","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.117Z","logger":"authentication-controller","msg":"Handling Integrated OAuth with OIDC configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"12570f3b-d442-4092-9967-ce649b1c4f8b","cluster":"raycluster-image-validation"} {"level":"error","ts":"2026-09-11T13:03:29.123Z","msg":"Failed to create CA Certificate","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f06dacc6-ec22-4827-b122-124fdaa2d27c","error":"certificates.cert-manager.io \"ray-ca-certificate-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).createCACertificate\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:890\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).reconcileRootCA\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:483\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:165\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:03:29.123Z","msg":"Failed to reconcile RootCA","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f06dacc6-ec22-4827-b122-124fdaa2d27c","error":"certificates.cert-manager.io \"ray-ca-certificate-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:166\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.124Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f06dacc6-ec22-4827-b122-124fdaa2d27c"} {"level":"error","ts":"2026-09-11T13:03:29.124Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"f06dacc6-ec22-4827-b122-124fdaa2d27c","error":"certificates.cert-manager.io \"ray-ca-certificate-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.144Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c38203a8-ac96-42a3-9b58-fa6fbb0ef2dc","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.144Z","msg":"Creating self-signed issuer for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c38203a8-ac96-42a3-9b58-fa6fbb0ef2dc","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.150Z","logger":"networkpolicy-controller","msg":"RayCluster is being deleted, NetworkPolicies will be garbage collected","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"a50202ca-1b58-4c28-923d-0b25cd8b03c9"} {"level":"info","ts":"2026-09-11T13:03:29.150Z","logger":"controllers.RayCluster","msg":"RayCluster is being deleted, just ignore","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"reconcileID":"2888f670-400c-4a6b-9d8e-1b1259e048f4"} {"level":"error","ts":"2026-09-11T13:03:29.150Z","logger":"authentication-controller","msg":"Failed to handle authentication configuration","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"12570f3b-d442-4092-9967-ce649b1c4f8b","error":"failed to ensure OIDC resources: failed to ensure ConfigMap: failed to create or update OIDC ConfigMap: configmaps \"kube-rbac-proxy-config-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:168\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.150Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"12570f3b-d442-4092-9967-ce649b1c4f8b"} {"level":"error","ts":"2026-09-11T13:03:29.150Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"12570f3b-d442-4092-9967-ce649b1c4f8b","error":"failed to ensure OIDC resources: failed to ensure ConfigMap: failed to create or update OIDC ConfigMap: configmaps \"kube-rbac-proxy-config-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.150Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"65bc887f-67c9-4b59-8217-5cd27d438776","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:03:29.150Z","logger":"authentication-controller","msg":"RayCluster is being deleted, running finalizer","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"65bc887f-67c9-4b59-8217-5cd27d438776","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.150Z","logger":"authentication-controller","msg":"Running authentication resource cleanup (finalizer)","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"65bc887f-67c9-4b59-8217-5cd27d438776","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.150Z","logger":"authentication-controller","msg":"ReferenceGrant cleanup check","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"65bc887f-67c9-4b59-8217-5cd27d438776","namespace":"test-ns-kppl8","otherAuthClusters":0,"action":"will delete"} {"level":"error","ts":"2026-09-11T13:03:29.158Z","msg":"Failed to create self-signed issuer","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c38203a8-ac96-42a3-9b58-fa6fbb0ef2dc","error":"issuers.cert-manager.io \"ray-selfsigned-issuer-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).createSelfSignedIssuer\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:552\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).reconcileSelfSignedIssuer\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:502\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:159\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:03:29.158Z","msg":"Failed to reconcile self-signed issuer","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c38203a8-ac96-42a3-9b58-fa6fbb0ef2dc","error":"issuers.cert-manager.io \"ray-selfsigned-issuer-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:160\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.158Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c38203a8-ac96-42a3-9b58-fa6fbb0ef2dc"} {"level":"error","ts":"2026-09-11T13:03:29.158Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"c38203a8-ac96-42a3-9b58-fa6fbb0ef2dc","error":"issuers.cert-manager.io \"ray-selfsigned-issuer-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.158Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"bc4d7aaf-f7c2-4db5-a321-325ddeef65e3","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.158Z","msg":"Creating self-signed issuer for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"bc4d7aaf-f7c2-4db5-a321-325ddeef65e3","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.166Z","logger":"authentication-controller","msg":"Deleted shared ReferenceGrant (last cluster with auth)","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"65bc887f-67c9-4b59-8217-5cd27d438776","name":"kuberay-gateway-access","namespace":"test-ns-kppl8"} {"level":"error","ts":"2026-09-11T13:03:29.171Z","msg":"Failed to create self-signed issuer","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"bc4d7aaf-f7c2-4db5-a321-325ddeef65e3","error":"issuers.cert-manager.io \"ray-selfsigned-issuer-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).createSelfSignedIssuer\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:552\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).reconcileSelfSignedIssuer\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:502\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:159\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:03:29.171Z","msg":"Failed to reconcile self-signed issuer","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"bc4d7aaf-f7c2-4db5-a321-325ddeef65e3","error":"issuers.cert-manager.io \"ray-selfsigned-issuer-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:160\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.171Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"bc4d7aaf-f7c2-4db5-a321-325ddeef65e3"} {"level":"error","ts":"2026-09-11T13:03:29.171Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"bc4d7aaf-f7c2-4db5-a321-325ddeef65e3","error":"issuers.cert-manager.io \"ray-selfsigned-issuer-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.183Z","logger":"authentication-controller","msg":"Deleted HTTPRoute","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"65bc887f-67c9-4b59-8217-5cd27d438776","name":"test-ns-kppl8-raycluster-image-validation","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-09-11T13:03:29.199Z","msg":"MTLS is enabled for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"94d49965-1307-433e-b691-b84cb1c27383","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.199Z","msg":"Creating self-signed issuer for RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"94d49965-1307-433e-b691-b84cb1c27383","rayCluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.200Z","logger":"authentication-controller","msg":"Deleted service account","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"65bc887f-67c9-4b59-8217-5cd27d438776","serviceAccount":"raycluster-image-validation-oauth-proxy-sa","mode":"IntegratedOAuth"} {"level":"error","ts":"2026-09-11T13:03:29.212Z","msg":"Failed to create self-signed issuer","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"94d49965-1307-433e-b691-b84cb1c27383","error":"issuers.cert-manager.io \"ray-selfsigned-issuer-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).createSelfSignedIssuer\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:552\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).reconcileSelfSignedIssuer\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:502\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:159\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-09-11T13:03:29.212Z","msg":"Failed to reconcile self-signed issuer","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"94d49965-1307-433e-b691-b84cb1c27383","error":"issuers.cert-manager.io \"ray-selfsigned-issuer-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayClusterMTLSController).Reconcile\n\t/workspace/controllers/ray/raycluster_mtls_controller.go:160\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.212Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"94d49965-1307-433e-b691-b84cb1c27383"} {"level":"error","ts":"2026-09-11T13:03:29.212Z","msg":"Reconciler error","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"94d49965-1307-433e-b691-b84cb1c27383","error":"issuers.cert-manager.io \"ray-selfsigned-issuer-raycluster-image-validation\" is forbidden: unable to create new content in namespace test-ns-kppl8 because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.217Z","logger":"raycluster-resource","msg":"default","name":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.217Z","logger":"raycluster-resource","msg":"enforcing secure trusted network on OpenShift","name":"raycluster-image-validation","namespace":"test-ns-kppl8"} {"level":"info","ts":"2026-09-11T13:03:29.238Z","logger":"authentication-controller","msg":"Authentication cleanup complete, finalizer removed","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"65bc887f-67c9-4b59-8217-5cd27d438776","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.238Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"233dd12e-e4dc-4706-afe0-5881a9830d51","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:03:29.238Z","logger":"authentication-controller","msg":"RayCluster is being deleted, running finalizer","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"233dd12e-e4dc-4706-afe0-5881a9830d51","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.238Z","logger":"authentication-controller","msg":"Running authentication resource cleanup (finalizer)","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"233dd12e-e4dc-4706-afe0-5881a9830d51","cluster":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.238Z","logger":"authentication-controller","msg":"ReferenceGrant cleanup check","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"233dd12e-e4dc-4706-afe0-5881a9830d51","namespace":"test-ns-kppl8","otherAuthClusters":0,"action":"will delete"} {"level":"info","ts":"2026-09-11T13:03:29.238Z","logger":"authentication-controller","msg":"ReferenceGrant already deleted","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"233dd12e-e4dc-4706-afe0-5881a9830d51","name":"kuberay-gateway-access"} {"level":"info","ts":"2026-09-11T13:03:29.238Z","msg":"Cleaning up MTLS resources for deleted RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"875c7d29-0fb3-44f4-9cdf-18edcbdc7530","namespace":"test-ns-kppl8","clusterName":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.238Z","logger":"networkpolicy-controller","msg":"RayCluster not found, NetworkPolicies will be garbage collected","controller":"networkpolicy","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"db4fdeca-af98-4cd7-b774-2d6b1e81ba26"} {"level":"error","ts":"2026-09-11T13:03:29.252Z","logger":"authentication-controller","msg":"Failed to remove finalizer","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"233dd12e-e4dc-4706-afe0-5881a9830d51","error":"rayclusters.ray.io \"raycluster-image-validation\" not found","stacktrace":"github.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).handleDeletion\n\t/workspace/controllers/ray/authentication_controller.go:196\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*AuthenticationController).Reconcile\n\t/workspace/controllers/ray/authentication_controller.go:137\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.252Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"233dd12e-e4dc-4706-afe0-5881a9830d51"} {"level":"error","ts":"2026-09-11T13:03:29.252Z","msg":"Reconciler error","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"233dd12e-e4dc-4706-afe0-5881a9830d51","error":"rayclusters.ray.io \"raycluster-image-validation\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-09-11T13:03:29.252Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"d5b1e2e7-e79c-48df-87d1-9ce6084240ba","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:03:29.252Z","logger":"authentication-controller","msg":"RayCluster not found, likely deleted - checking for orphaned resources","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"d5b1e2e7-e79c-48df-87d1-9ce6084240ba"} {"level":"info","ts":"2026-09-11T13:03:29.252Z","logger":"authentication-controller","msg":"Checking for orphaned ReferenceGrant","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"d5b1e2e7-e79c-48df-87d1-9ce6084240ba","namespace":"test-ns-kppl8","authClusters":0} {"level":"info","ts":"2026-09-11T13:03:29.252Z","logger":"authentication-controller","msg":"No orphaned ReferenceGrant found","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"d5b1e2e7-e79c-48df-87d1-9ce6084240ba","namespace":"test-ns-kppl8"} {"level":"info","ts":"2026-09-11T13:03:29.257Z","logger":"authentication-controller","msg":"Reconciling Authentication","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4f14ebf5-448f-4a92-81e7-3a49cad0b028","namespacedName":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"}} {"level":"info","ts":"2026-09-11T13:03:29.257Z","logger":"authentication-controller","msg":"RayCluster not found, likely deleted - checking for orphaned resources","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4f14ebf5-448f-4a92-81e7-3a49cad0b028"} {"level":"info","ts":"2026-09-11T13:03:29.257Z","logger":"authentication-controller","msg":"Checking for orphaned ReferenceGrant","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4f14ebf5-448f-4a92-81e7-3a49cad0b028","namespace":"test-ns-kppl8","authClusters":0} {"level":"info","ts":"2026-09-11T13:03:29.257Z","logger":"authentication-controller","msg":"No orphaned ReferenceGrant found","controller":"authentication","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"4f14ebf5-448f-4a92-81e7-3a49cad0b028","namespace":"test-ns-kppl8"} {"level":"info","ts":"2026-09-11T13:03:29.335Z","msg":"Successfully cleaned up MTLS resources","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"875c7d29-0fb3-44f4-9cdf-18edcbdc7530","namespace":"test-ns-kppl8","clusterName":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.372Z","msg":"Cleaning up MTLS resources for deleted RayCluster","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"d4194a98-ff63-4467-a5ff-6d8e4cd8cf2b","namespace":"test-ns-kppl8","clusterName":"raycluster-image-validation"} {"level":"info","ts":"2026-09-11T13:03:29.372Z","msg":"Successfully cleaned up MTLS resources","controller":"raycluster-mtls","controllerGroup":"ray.io","controllerKind":"RayCluster","RayCluster":{"name":"raycluster-image-validation","namespace":"test-ns-kppl8"},"namespace":"test-ns-kppl8","name":"raycluster-image-validation","reconcileID":"d4194a98-ff63-4467-a5ff-6d8e4cd8cf2b","namespace":"test-ns-kppl8","clusterName":"raycluster-image-validation"} ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-7cb4dd5dc6-jxrxf.log ==================== W0911 12:38:52.011040 1 connect.go:86] Retrying connection to PostgreSQL (attempt 1/25): failed to connect to `user=catalog_user database=model_catalog`: 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused W0911 12:38:53.013822 1 connect.go:86] Retrying connection to PostgreSQL (attempt 2/25): failed to connect to `user=catalog_user database=model_catalog`: 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused W0911 12:38:55.016675 1 connect.go:86] Retrying connection to PostgreSQL (attempt 3/25): failed to connect to `user=catalog_user database=model_catalog`: 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused W0911 12:38:58.018950 1 connect.go:86] Retrying connection to PostgreSQL (attempt 4/25): failed to connect to `user=catalog_user database=model_catalog`: 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused W0911 12:39:02.022688 1 connect.go:86] Retrying connection to PostgreSQL (attempt 5/25): failed to connect to `user=catalog_user database=model_catalog`: 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused W0911 12:39:07.026488 1 connect.go:86] Retrying connection to PostgreSQL (attempt 6/25): failed to connect to `user=catalog_user database=model_catalog`: 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused 172.31.148.124:5432 (model-catalog-postgres): dial error: dial tcp 172.31.148.124:5432: connect: connection refused I0911 12:39:13.038500 1 connect.go:94] Successfully connected to PostgreSQL database I0911 12:39:13.038525 1 catalog.go:137] Leader election configured: lock duration=1m0s, heartbeat=15s I0911 12:39:13.372803 1 service.go:131] Connecting to EmbedMD service... I0911 12:39:13.372828 1 service.go:138] Connected to EmbedMD service I0911 12:39:13.372832 1 repos.go:35] Getting types... I0911 12:39:13.372856 1 election.go:289] Attempting to acquire leadership... W0911 12:39:13.383955 1 service.go:187] Unable to get types: no types available: ERROR: relation "Type" does not exist (SQLSTATE 42P01) I0911 12:39:13.392830 1 election.go:357] Successfully acquired leadership lock: catalog-leader I0911 12:39:13.435526 1 service.go:163] Running migrations... I0911 12:39:14.384341 1 repos.go:35] Getting types... W0911 12:39:14.385191 1 service.go:187] Unable to get types: no types available: ERROR: relation "Type" does not exist (SQLSTATE 42P01) I0911 12:39:15.288443 1 service.go:168] Migrations completed I0911 12:39:15.288471 1 service.go:170] Syncing types... I0911 12:39:15.385421 1 repos.go:35] Getting types... I0911 12:39:15.395780 1 repos.go:47] Types retrieved I0911 12:39:15.395802 1 repos.go:63] All required types validated successfully 2026/09/11 12:39:15 INFO initializing plugin plugin=agent version=v1alpha1 basePath=/api/agent_catalog/v1alpha1 2026/09/11 12:39:15 INFO initializing plugin plugin=mcp version=v1alpha1 basePath=/api/mcp_catalog/v1alpha1 2026/09/11 12:39:15 INFO initializing plugin plugin=model version=v1alpha1 basePath=/api/model_catalog/v1alpha1 I0911 12:39:15.395969 1 performance_metrics.go:210] Loading performance metrics data from [/shared-benchmark-data] I0911 12:39:15.403795 1 performance_metrics.go:295] Built model directory cache (131 models indexed) 2026/09/11 12:39:15 INFO initializing plugin plugin=skill version=v1alpha1 basePath=/api/skill_catalog/v1alpha1 2026/09/11 12:39:15 INFO all plugins initialized count=4 2026/09/11 12:39:15 INFO mounting plugin routes plugin=agent 2026/09/11 12:39:15 INFO mounting plugin routes plugin=mcp 2026/09/11 12:39:15 INFO mounting plugin routes plugin=model 2026/09/11 12:39:15 INFO mounting plugin routes plugin=skill 2026/09/11 12:39:15 INFO starting plugin plugin=agent I0911 12:39:15.404011 1 base.go:58] Starting agent plugin in read-only mode (standby) I0911 12:39:15.404019 1 loader.go:48] Initializing agent loader - parsing configs W0911 12:39:15.404684 1 config.go:24] Configuration file /data/default-sources/sources.yaml uses deprecated 'catalogs' field. Please rename to 'model_catalogs'. I0911 12:39:15.404702 1 loader.go:251] reading agent catalog config type yaml... I0911 12:39:15.404708 1 loader.go:261] loaded agent source rh_agents of type yaml I0911 12:39:15.404822 1 loader.go:56] agent loader config parsing complete I0911 12:39:15.404840 1 base.go:73] agent plugin read-only mode initialized 2026/09/11 12:39:15 INFO starting plugin plugin=mcp I0911 12:39:15.404852 1 base.go:58] Starting mcp plugin in read-only mode (standby) I0911 12:39:15.404858 1 loader.go:85] Initializing MCP loader - parsing configs W0911 12:39:15.405411 1 config.go:24] Configuration file /data/default-sources/sources.yaml uses deprecated 'catalogs' field. Please rename to 'model_catalogs'. I0911 12:39:15.405426 1 loader.go:154] reading MCP catalog config type yaml... I0911 12:39:15.405431 1 loader.go:170] loaded MCP source rh_mcp_servers of type yaml I0911 12:39:15.405434 1 loader.go:154] reading MCP catalog config type yaml... I0911 12:39:15.405438 1 loader.go:170] loaded MCP source rh_partner_mcp_servers of type yaml I0911 12:39:15.405441 1 loader.go:154] reading MCP catalog config type yaml... I0911 12:39:15.405444 1 loader.go:170] loaded MCP source community_mcp_servers of type yaml I0911 12:39:15.405581 1 loader.go:93] MCP loader config parsing complete I0911 12:39:15.405592 1 base.go:73] mcp plugin read-only mode initialized 2026/09/11 12:39:15 INFO starting plugin plugin=model I0911 12:39:15.405603 1 base.go:58] Starting model plugin in read-only mode (standby) W0911 12:39:15.406312 1 config.go:24] Configuration file /data/default-sources/sources.yaml uses deprecated 'catalogs' field. Please rename to 'model_catalogs'. I0911 12:39:15.406350 1 loader.go:235] reading config type yaml... I0911 12:39:15.406357 1 loader.go:253] loaded source redhat_ai_models of type yaml I0911 12:39:15.406360 1 loader.go:235] reading config type yaml... I0911 12:39:15.406364 1 loader.go:253] loaded source redhat_ai_validated_models of type yaml I0911 12:39:15.406367 1 loader.go:235] reading config type yaml... I0911 12:39:15.406370 1 loader.go:253] loaded source other_models of type yaml I0911 12:39:15.406616 1 service.go:175] Syncing types completed I0911 12:39:15.406688 1 base.go:73] model plugin read-only mode initialized 2026/09/11 12:39:15 INFO starting plugin plugin=skill I0911 12:39:15.406740 1 base.go:58] Starting skill plugin in read-only mode (standby) I0911 12:39:15.406750 1 loader.go:212] Initializing skill loader - parsing configs W0911 12:39:15.407626 1 config.go:24] Configuration file /data/default-sources/sources.yaml uses deprecated 'catalogs' field. Please rename to 'model_catalogs'. I0911 12:39:15.407767 1 loader.go:218] skill loader config parsing complete I0911 12:39:15.407788 1 base.go:73] skill plugin read-only mode initialized I0911 12:39:15.407832 1 repos.go:35] Getting types... I0911 12:39:15.407919 1 catalog.go:223] Catalog API server listening on 0.0.0.0:8080 I0911 12:39:15.408693 1 repos.go:47] Types retrieved I0911 12:39:15.408710 1 repos.go:63] All required types validated successfully 2026/09/11 12:39:15 INFO plugin becoming leader plugin=skill 2026/09/11 12:39:15 INFO plugin becoming leader plugin=agent I0911 12:39:15.408792 1 base.go:118] skill plugin becoming leader I0911 12:39:15.408796 1 base.go:118] agent plugin becoming leader 2026/09/11 12:39:15 INFO plugin becoming leader plugin=mcp I0911 12:39:15.408812 1 base.go:118] mcp plugin becoming leader 2026/09/11 12:39:15 INFO plugin becoming leader plugin=model I0911 12:39:15.408837 1 loader.go:61] agent loader performing leader operations I0911 12:39:15.408837 1 loader.go:223] skill loader performing leader operations I0911 12:39:15.408838 1 base.go:118] model plugin becoming leader I0911 12:39:15.408846 1 loader_base.go:88] Waiting for inflight writes to complete... I0911 12:39:15.408847 1 loader_base.go:88] Waiting for inflight writes to complete... I0911 12:39:15.408860 1 loader_base.go:98] All inflight writes completed I0911 12:39:15.408864 1 loader_base.go:98] All inflight writes completed I0911 12:39:15.408869 1 loader.go:104] MCP loader performing leader operations I0911 12:39:15.408871 1 loader_base.go:88] Waiting for inflight writes to complete... I0911 12:39:15.408878 1 loader_base.go:88] Waiting for inflight writes to complete... I0911 12:39:15.408891 1 loader_base.go:98] All inflight writes completed I0911 12:39:15.408893 1 loader_base.go:98] All inflight writes completed I0911 12:39:15.411041 1 loader.go:409] Reading models from yaml source redhat_ai_models I0911 12:39:15.411066 1 loader.go:409] Reading models from yaml source redhat_ai_validated_models I0911 12:39:15.411100 1 loader.go:409] Reading models from yaml source other_models I0911 12:39:15.411113 1 loader_base.go:88] Waiting for inflight writes to complete... I0911 12:39:15.411872 1 loader.go:91] agent loader leader operations launched I0911 12:39:15.411898 1 loader_base.go:88] Waiting for inflight writes to complete... I0911 12:39:15.424078 1 loader.go:250] Loading MCP servers from source: Community MCP Servers (id: community_mcp_servers) I0911 12:39:15.424177 1 loader.go:250] Loading MCP servers from source: Red Hat MCP Servers (id: rh_mcp_servers) I0911 12:39:15.424196 1 loader.go:250] Loading MCP servers from source: Red Hat Partner MCP Servers (id: rh_partner_mcp_servers) I0911 12:39:15.424204 1 loader.go:118] MCP loader leader operations launched I0911 12:39:15.424221 1 loader_base.go:88] Waiting for inflight writes to complete... I0911 12:39:15.427522 1 loader.go:289] skill loader leader operations launched I0911 12:39:15.427602 1 loader_base.go:88] Waiting for inflight writes to complete... I0911 12:39:15.427655 1 loader_base.go:98] All inflight writes completed I0911 12:39:15.427684 1 base.go:139] skill plugin leader mode active I0911 12:39:15.446468 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w4a16 with 1 artifact(s) I0911 12:39:15.582751 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 with 1 artifact(s) I0911 12:39:15.611871 1 loader.go:315] Loading model other_models:Granite Guardian 3.2 5B with 1 artifact(s) I0911 12:39:15.631802 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Apertus-8B-Instruct-2509-FP8-dynamic with 1 artifact(s) I0911 12:39:15.696381 1 loader.go:329] community_mcp_servers: loaded 2 MCP servers I0911 12:39:15.783006 1 loader.go:158] rh_agents: loaded 14 agents I0911 12:39:15.893622 1 loader_base.go:98] All inflight writes completed I0911 12:39:15.893645 1 base.go:139] agent plugin leader mode active I0911 12:39:16.109369 1 performance_metrics.go:329] Loaded 117 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Apertus-8B-Instruct-2509-FP8-dynamic I0911 12:39:16.109412 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/Mistral-Small-24B-Instruct-2501-FP8-dynamic with 1 artifact(s) I0911 12:39:16.138875 1 loader.go:315] Loading model other_models:Llama-3.1-Nemotron-Safety-Guard-8B-v3 with 1 artifact(s) I0911 12:39:16.147763 1 loader.go:329] rh_mcp_servers: loaded 3 MCP servers I0911 12:39:16.188049 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/DeepSeek-R1-0528-quantized.w4a16 with 1 artifact(s) I0911 12:39:16.308698 1 performance_metrics.go:329] Loaded 55 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/DeepSeek-R1-0528-quantized.w4a16 I0911 12:39:16.308730 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w4a16 with 1 artifact(s) I0911 12:39:16.330076 1 loader.go:315] Loading model other_models:Magistral Small 1.2 with 1 artifact(s) I0911 12:39:16.348174 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Devstral-Small-2-24B-Instruct-2512 with 1 artifact(s) I0911 12:39:16.502840 1 loader.go:329] rh_partner_mcp_servers: loaded 5 MCP servers I0911 12:39:16.510205 1 loader_base.go:98] All inflight writes completed I0911 12:39:16.510228 1 base.go:139] mcp plugin leader mode active I0911 12:39:16.630486 1 performance_metrics.go:329] Loaded 138 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Devstral-Small-2-24B-Instruct-2512 I0911 12:39:16.630532 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w8a8 with 1 artifact(s) I0911 12:39:16.647894 1 loader.go:315] Loading model other_models:Molmo2-4B with 1 artifact(s) I0911 12:39:16.662701 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/GLM-5.2-FP8 with 1 artifact(s) I0911 12:39:16.707481 1 performance_metrics.go:329] Loaded 15 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/GLM-5.2-FP8 I0911 12:39:16.707516 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/Voxtral-Mini-3B-2507-FP8-dynamic with 1 artifact(s) I0911 12:39:16.725959 1 loader.go:315] Loading model other_models:RedHatAI/Laguna-XS.2 with 1 artifact(s) I0911 12:39:16.750736 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Kimi-K2-Instruct-quantized.w4a16 with 1 artifact(s) I0911 12:39:16.811272 1 performance_metrics.go:329] Loaded 22 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Kimi-K2-Instruct-quantized.w4a16 I0911 12:39:16.811306 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/gemma-2-9b-it with 1 artifact(s) I0911 12:39:16.827864 1 loader.go:315] Loading model other_models:RedHatAI/Laguna-XS.2-FP8 with 1 artifact(s) I0911 12:39:16.854081 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.1-8B-Instruct with 1 artifact(s) I0911 12:39:17.038293 1 performance_metrics.go:329] Loaded 107 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.1-8B-Instruct I0911 12:39:17.038344 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/gemma-2-9b-it-FP8 with 1 artifact(s) I0911 12:39:17.055511 1 loader.go:315] Loading model other_models:RedHatAI/Laguna-XS.2-NVFP4 with 1 artifact(s) I0911 12:39:17.094696 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.1-Nemotron-70B-Instruct-HF with 1 artifact(s) I0911 12:39:17.169795 1 performance_metrics.go:329] Loaded 30 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.1-Nemotron-70B-Instruct-HF I0911 12:39:17.169828 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/granite-3.1-8b-base-quantized.w4a16 with 1 artifact(s) I0911 12:39:17.187860 1 loader.go:315] Loading model other_models:RedHatAI/Llama-3.1-8B-Instruct-essential with 1 artifact(s) I0911 12:39:17.207766 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.1-Nemotron-70B-Instruct-HF-FP8-dynamic with 1 artifact(s) I0911 12:39:17.284967 1 performance_metrics.go:329] Loaded 35 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.1-Nemotron-70B-Instruct-HF-FP8-dynamic I0911 12:39:17.285004 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/granite-3.3-8b-instruct with 1 artifact(s) I0911 12:39:17.301571 1 loader.go:315] Loading model other_models:RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8 with 1 artifact(s) I0911 12:39:17.333777 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct with 1 artifact(s) I0911 12:39:17.407622 1 performance_metrics.go:329] Loaded 35 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct I0911 12:39:17.407658 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/granite-4.0-h-small with 1 artifact(s) I0911 12:39:17.422639 1 loader.go:315] Loading model other_models:RedHatAI/Qwen2.5-1.5B-quantized.w8a8 with 1 artifact(s) I0911 12:39:17.450514 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic with 1 artifact(s) I0911 12:39:17.536459 1 performance_metrics.go:329] Loaded 36 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic I0911 12:39:17.536490 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/granite-4.0-h-tiny with 1 artifact(s) I0911 12:39:17.551996 1 loader.go:315] Loading model other_models:RedHatAI/Qwen3-8B with 1 artifact(s) I0911 12:39:17.582465 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-quantized.w4a16 with 1 artifact(s) I0911 12:39:17.657980 1 performance_metrics.go:329] Loaded 30 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-quantized.w4a16 I0911 12:39:17.658019 1 loader.go:315] Loading model redhat_ai_models:RedHatAI/whisper-large-v3-turbo-quantized.w4a16 with 1 artifact(s) I0911 12:39:17.675403 1 loader.go:315] Loading model other_models:RedHatAI/Qwen3-8B-speculator.dflash with 1 artifact(s) I0911 12:39:17.704500 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-quantized.w8a8 with 1 artifact(s) I0911 12:39:17.804455 1 performance_metrics.go:329] Loaded 43 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-quantized.w8a8 I0911 12:39:17.804490 1 loader.go:315] Loading model redhat_ai_models:granite-7b-redhat-lab with 1 artifact(s) I0911 12:39:17.818640 1 loader.go:315] Loading model other_models:RedHatAI/Qwen3-Embedding-8B with 1 artifact(s) I0911 12:39:17.850755 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Maverick-17B-128E-Instruct with 1 artifact(s) I0911 12:39:17.894252 1 performance_metrics.go:329] Loaded 3 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Maverick-17B-128E-Instruct I0911 12:39:17.894281 1 loader.go:315] Loading model redhat_ai_models:granite-8b-starter-v1 with 1 artifact(s) I0911 12:39:17.908787 1 loader.go:315] Loading model other_models:RedHatAI/all-MiniLM-L6-v2 with 1 artifact(s) I0911 12:39:17.926028 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Maverick-17B-128E-Instruct-FP8 with 1 artifact(s) I0911 12:39:17.990345 1 performance_metrics.go:329] Loaded 18 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Maverick-17B-128E-Instruct-FP8 I0911 12:39:17.990377 1 loader.go:315] Loading model redhat_ai_models:granite-8b-lab-v1 with 1 artifact(s) I0911 12:39:18.005072 1 loader.go:315] Loading model other_models:RedHatAI/bge-m3 with 1 artifact(s) I0911 12:39:18.022034 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct with 1 artifact(s) I0911 12:39:18.081148 1 performance_metrics.go:329] Loaded 15 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct I0911 12:39:18.081293 1 loader.go:315] Loading model redhat_ai_models:granite-3.1-8b-lab-v1 with 1 artifact(s) I0911 12:39:18.097960 1 loader.go:315] Loading model other_models:RedHatAI/embeddinggemma-300m with 1 artifact(s) I0911 12:39:18.119967 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic with 1 artifact(s) I0911 12:39:18.211091 1 performance_metrics.go:329] Loaded 40 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic I0911 12:39:18.211130 1 loader.go:315] Loading model redhat_ai_models:granite-8b-code-instruct with 1 artifact(s) I0911 12:39:18.225956 1 loader.go:315] Loading model other_models:RedHatAI/gemma-3-12b-it with 1 artifact(s) I0911 12:39:18.267130 1 performance_metrics.go:329] Loaded 6 performance metrics artifacts for model other_models:RedHatAI/gemma-3-12b-it I0911 12:39:18.267165 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16 with 1 artifact(s) I0911 12:39:18.338287 1 performance_metrics.go:329] Loaded 16 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16 I0911 12:39:18.338348 1 loader.go:315] Loading model redhat_ai_models:granite-8b-code-base with 1 artifact(s) I0911 12:39:18.353360 1 loader.go:315] Loading model other_models:RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3 with 1 artifact(s) I0911 12:39:18.382639 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic with 1 artifact(s) I0911 12:39:18.430925 1 performance_metrics.go:329] Loaded 3 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic I0911 12:39:18.430961 1 loader.go:315] Loading model redhat_ai_models:prometheus-8x7b-v2-0 with 1 artifact(s) I0911 12:39:18.430974 1 loader.go:453] redhat_ai_models: loaded 20 models I0911 12:39:18.444303 1 loader.go:470] redhat_ai_models: cleaned up 0 models I0911 12:39:18.449487 1 loader.go:315] Loading model other_models:RedHatAI/gemma-4-31B-it-speculator.dflash with 1 artifact(s) I0911 12:39:18.479546 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/MiniMax-M2.5 with 1 artifact(s) I0911 12:39:18.572546 1 performance_metrics.go:329] Loaded 41 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/MiniMax-M2.5 I0911 12:39:18.572591 1 loader.go:315] Loading model other_models:RedHatAI/gemma-4-31B-it-speculator.eagle3 with 1 artifact(s) I0911 12:39:18.607464 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Ministral-3-14B-Instruct-2512 with 1 artifact(s) I0911 12:39:18.819090 1 performance_metrics.go:329] Loaded 130 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Ministral-3-14B-Instruct-2512 I0911 12:39:18.819128 1 loader.go:315] Loading model other_models:RedHatAI/granite-embedding-english-r2 with 1 artifact(s) I0911 12:39:18.836843 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Ministral-3-3B-Instruct-2512 with 1 artifact(s) I0911 12:39:19.092739 1 performance_metrics.go:329] Loaded 149 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Ministral-3-3B-Instruct-2512 I0911 12:39:19.092784 1 loader.go:315] Loading model other_models:RedHatAI/gte-Qwen2-7B-instruct with 1 artifact(s) I0911 12:39:19.111184 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Large-3-675B-Instruct-2512 with 1 artifact(s) I0911 12:39:19.177469 1 performance_metrics.go:329] Loaded 18 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Large-3-675B-Instruct-2512 I0911 12:39:19.177518 1 loader.go:315] Loading model other_models:RedHatAI/multilingual-e5-large with 1 artifact(s) I0911 12:39:19.195547 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4 with 1 artifact(s) I0911 12:39:19.317119 1 performance_metrics.go:329] Loaded 42 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4 I0911 12:39:19.317160 1 loader.go:315] Loading model other_models:RedHatAI/nomic-embed-text-v1.5 with 1 artifact(s) I0911 12:39:19.334406 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Medium-3.5-128B with 1 artifact(s) I0911 12:39:19.448989 1 performance_metrics.go:329] Loaded 69 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Medium-3.5-128B I0911 12:39:19.449022 1 loader.go:315] Loading model other_models:RedHatAI/snowflake-arctic-embed-l-v2.0 with 1 artifact(s) I0911 12:39:19.510922 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-24B-Instruct-2501 with 1 artifact(s) 2026/09/11 12:39:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35736 - 200 81B in 42.143µs I0911 12:39:19.640204 1 performance_metrics.go:329] Loaded 81 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-24B-Instruct-2501 I0911 12:39:19.640239 1 loader.go:315] Loading model other_models:google/gemma-4-26B-A4B with 1 artifact(s) I0911 12:39:19.659180 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503 with 1 artifact(s) I0911 12:39:19.837214 1 performance_metrics.go:329] Loaded 105 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503 I0911 12:39:19.837254 1 loader.go:315] Loading model other_models:google/gemma-4-31B with 1 artifact(s) I0911 12:39:19.857055 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic with 1 artifact(s) I0911 12:39:20.039426 1 performance_metrics.go:329] Loaded 112 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic I0911 12:39:20.039465 1 loader.go:315] Loading model other_models:granite-docling-258m with 1 artifact(s) I0911 12:39:20.066129 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16 with 1 artifact(s) I0911 12:39:20.351297 1 performance_metrics.go:329] Loaded 104 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16 I0911 12:39:20.351352 1 loader.go:315] Loading model other_models:olmOCR-2-7B-1025-FP8 with 1 artifact(s) I0911 12:39:20.394522 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w8a8 with 1 artifact(s) I0911 12:39:20.609160 1 performance_metrics.go:329] Loaded 123 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w8a8 I0911 12:39:20.609202 1 loader.go:315] Loading model other_models:openai/gpt-oss-safeguard-120b with 1 artifact(s) I0911 12:39:20.645189 1 performance_metrics.go:329] Loaded 6 performance metrics artifacts for model other_models:openai/gpt-oss-safeguard-120b I0911 12:39:20.645229 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-4-119B-2603 with 1 artifact(s) I0911 12:39:20.749412 1 performance_metrics.go:329] Loaded 40 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-4-119B-2603 I0911 12:39:20.749455 1 loader.go:315] Loading model other_models:openai/gpt-oss-safeguard-20b with 1 artifact(s) I0911 12:39:20.785755 1 performance_metrics.go:329] Loaded 6 performance metrics artifacts for model other_models:openai/gpt-oss-safeguard-20b I0911 12:39:20.785782 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-4-119B-2603-NVFP4 with 1 artifact(s) I0911 12:39:20.947451 1 performance_metrics.go:329] Loaded 91 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-4-119B-2603-NVFP4 I0911 12:39:20.947486 1 loader.go:315] Loading model other_models:openai/whisper-large-v3 with 1 artifact(s) I0911 12:39:21.010443 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Mixtral-8x7B-Instruct-v0.1 with 1 artifact(s) I0911 12:39:21.117769 1 performance_metrics.go:329] Loaded 59 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mixtral-8x7B-Instruct-v0.1 I0911 12:39:21.117805 1 loader.go:315] Loading model other_models:poolside/Laguna-XS.2-speculator.dflash with 1 artifact(s) I0911 12:39:21.117884 1 loader.go:453] other_models: loaded 33 models I0911 12:39:21.134157 1 loader.go:470] other_models: cleaned up 0 models I0911 12:39:21.135618 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 with 1 artifact(s) I0911 12:39:21.404781 1 performance_metrics.go:329] Loaded 126 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 I0911 12:39:21.404831 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 with 1 artifact(s) I0911 12:39:21.536181 1 performance_metrics.go:329] Loaded 80 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 I0911 12:39:21.536215 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 with 1 artifact(s) I0911 12:39:21.699201 1 performance_metrics.go:329] Loaded 79 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 I0911 12:39:21.699236 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 with 1 artifact(s) I0911 12:39:21.786831 1 performance_metrics.go:329] Loaded 31 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 I0911 12:39:21.786874 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block with 1 artifact(s) I0911 12:39:21.807667 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic with 1 artifact(s) I0911 12:39:21.857214 1 performance_metrics.go:329] Loaded 8 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic I0911 12:39:21.857246 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 with 1 artifact(s) I0911 12:39:21.922197 1 performance_metrics.go:329] Loaded 8 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 I0911 12:39:21.922229 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16 with 1 artifact(s) I0911 12:39:21.945522 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic with 1 artifact(s) I0911 12:39:22.051554 1 performance_metrics.go:329] Loaded 46 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic I0911 12:39:22.051590 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Phi-4-mini-instruct-FP8-dynamic with 1 artifact(s) I0911 12:39:22.228859 1 performance_metrics.go:329] Loaded 110 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Phi-4-mini-instruct-FP8-dynamic I0911 12:39:22.229040 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Phi-4-reasoning-FP8-dynamic with 1 artifact(s) I0911 12:39:22.400246 1 performance_metrics.go:329] Loaded 75 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Phi-4-reasoning-FP8-dynamic I0911 12:39:22.400302 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct with 1 artifact(s) I0911 12:39:22.600726 1 performance_metrics.go:329] Loaded 102 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct I0911 12:39:22.600767 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-FP8-dynamic with 1 artifact(s) I0911 12:39:22.805461 1 performance_metrics.go:329] Loaded 110 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-FP8-dynamic I0911 12:39:22.805515 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-quantized.w4a16 with 1 artifact(s) I0911 12:39:22.993801 1 performance_metrics.go:329] Loaded 87 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-quantized.w4a16 I0911 12:39:22.993838 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-quantized.w8a8 with 1 artifact(s) I0911 12:39:23.220567 1 performance_metrics.go:329] Loaded 106 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-quantized.w8a8 I0911 12:39:23.220614 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-8B-FP8-dynamic with 1 artifact(s) I0911 12:39:23.505024 1 performance_metrics.go:329] Loaded 155 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-8B-FP8-dynamic I0911 12:39:23.505063 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-Coder-480B-A35B-Instruct-FP8 with 1 artifact(s) I0911 12:39:23.569327 1 performance_metrics.go:329] Loaded 13 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-Coder-480B-A35B-Instruct-FP8 I0911 12:39:23.569373 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-Coder-Next-NVFP4 with 1 artifact(s) I0911 12:39:23.787464 1 performance_metrics.go:329] Loaded 119 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-Coder-Next-NVFP4 I0911 12:39:23.787506 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-Next-80B-A3B-Instruct-quantized.w4a16 with 1 artifact(s) I0911 12:39:24.047105 1 performance_metrics.go:329] Loaded 133 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-Next-80B-A3B-Instruct-quantized.w4a16 I0911 12:39:24.047151 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-VL-235B-A22B-Instruct-NVFP4 with 1 artifact(s) I0911 12:39:24.232009 1 performance_metrics.go:329] Loaded 79 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-VL-235B-A22B-Instruct-NVFP4 I0911 12:39:24.232042 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-VL-30B-A3B-Instruct with 1 artifact(s) I0911 12:39:24.491244 1 performance_metrics.go:329] Loaded 106 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-VL-30B-A3B-Instruct I0911 12:39:24.491288 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic with 1 artifact(s) 2026/09/11 12:39:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:50120 - 200 81B in 40.417µs I0911 12:39:24.588547 1 performance_metrics.go:329] Loaded 42 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic I0911 12:39:24.588579 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic with 1 artifact(s) I0911 12:39:24.786719 1 performance_metrics.go:329] Loaded 108 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic I0911 12:39:24.786760 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic with 1 artifact(s) I0911 12:39:24.873141 1 performance_metrics.go:329] Loaded 25 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic I0911 12:39:24.873178 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.6-35B-A3B with 1 artifact(s) I0911 12:39:25.136388 1 performance_metrics.go:329] Loaded 114 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.6-35B-A3B I0911 12:39:25.136433 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.6-35B-A3B-FP8 with 1 artifact(s) I0911 12:39:25.338505 1 performance_metrics.go:329] Loaded 104 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.6-35B-A3B-FP8 I0911 12:39:25.338540 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic with 1 artifact(s) I0911 12:39:25.614240 1 performance_metrics.go:329] Loaded 153 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic I0911 12:39:25.614282 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.6-35B-A3B-NVFP4 with 1 artifact(s) I0911 12:39:25.884672 1 performance_metrics.go:329] Loaded 146 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.6-35B-A3B-NVFP4 I0911 12:39:25.884712 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic with 1 artifact(s) I0911 12:39:25.930420 1 performance_metrics.go:329] Loaded 6 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic I0911 12:39:25.930457 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/diffusiongemma-26B-A4B-it-NVFP4 with 1 artifact(s) I0911 12:39:25.974239 1 performance_metrics.go:329] Loaded 6 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/diffusiongemma-26B-A4B-it-NVFP4 I0911 12:39:25.974264 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-3n-E4B-it-FP8-dynamic with 1 artifact(s) I0911 12:39:26.120938 1 performance_metrics.go:329] Loaded 81 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-3n-E4B-it-FP8-dynamic I0911 12:39:26.120974 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-12B-it-FP8-Dynamic with 1 artifact(s) I0911 12:39:26.343643 1 performance_metrics.go:329] Loaded 134 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-12B-it-FP8-Dynamic I0911 12:39:26.343684 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-12B-it-NVFP4 with 1 artifact(s) I0911 12:39:26.593906 1 performance_metrics.go:329] Loaded 120 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-12B-it-NVFP4 I0911 12:39:26.593949 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-26B-A4B-it with 1 artifact(s) I0911 12:39:26.813877 1 performance_metrics.go:329] Loaded 129 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-26B-A4B-it I0911 12:39:26.813922 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic with 1 artifact(s) I0911 12:39:27.029976 1 performance_metrics.go:329] Loaded 99 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic I0911 12:39:27.030025 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-26B-A4B-it-NVFP4 with 1 artifact(s) I0911 12:39:27.214962 1 performance_metrics.go:329] Loaded 102 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-26B-A4B-it-NVFP4 I0911 12:39:27.215010 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-31B-it with 1 artifact(s) I0911 12:39:27.320130 1 performance_metrics.go:329] Loaded 49 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-31B-it I0911 12:39:27.320171 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-31B-it-FP8-block with 1 artifact(s) I0911 12:39:27.437729 1 performance_metrics.go:329] Loaded 63 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-31B-it-FP8-block I0911 12:39:27.437769 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-31B-it-FP8-dynamic with 1 artifact(s) I0911 12:39:27.635919 1 performance_metrics.go:329] Loaded 110 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-31B-it-FP8-dynamic I0911 12:39:27.635959 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-31B-it-NVFP4 with 1 artifact(s) I0911 12:39:27.801009 1 performance_metrics.go:329] Loaded 72 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-31B-it-NVFP4 I0911 12:39:27.801067 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gemma-4-E4B-it with 1 artifact(s) I0911 12:39:28.041236 1 performance_metrics.go:329] Loaded 144 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-4-E4B-it I0911 12:39:28.041270 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gpt-oss-120b with 1 artifact(s) I0911 12:39:28.321248 1 performance_metrics.go:329] Loaded 148 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gpt-oss-120b I0911 12:39:28.321289 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gpt-oss-120b-essential with 1 artifact(s) I0911 12:39:28.604218 1 performance_metrics.go:329] Loaded 148 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gpt-oss-120b-essential I0911 12:39:28.604258 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gpt-oss-20b with 1 artifact(s) I0911 12:39:28.946457 1 performance_metrics.go:329] Loaded 183 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gpt-oss-20b I0911 12:39:28.946496 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/gpt-oss-20b-essential with 1 artifact(s) I0911 12:39:29.250307 1 performance_metrics.go:329] Loaded 177 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gpt-oss-20b-essential I0911 12:39:29.250370 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct with 1 artifact(s) I0911 12:39:29.442448 1 performance_metrics.go:329] Loaded 95 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct I0911 12:39:29.442493 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-FP8-dynamic with 1 artifact(s) I0911 12:39:29.535590 1 performance_metrics.go:329] Loaded 6 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-FP8-dynamic I0911 12:39:29.535625 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-quantized.w4a16 with 1 artifact(s) 2026/09/11 12:39:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:50126 - 200 81B in 46.314µs I0911 12:39:29.591062 1 performance_metrics.go:329] Loaded 4 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-quantized.w4a16 I0911 12:39:29.591097 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-quantized.w8a8 with 1 artifact(s) I0911 12:39:29.641969 1 performance_metrics.go:329] Loaded 1 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-quantized.w8a8 I0911 12:39:29.642004 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/granite-4.0-h-small-FP8-dynamic with 1 artifact(s) I0911 12:39:29.741458 1 performance_metrics.go:329] Loaded 42 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-4.0-h-small-FP8-dynamic I0911 12:39:29.741505 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/granite-4.0-h-tiny-FP8-dynamic with 1 artifact(s) I0911 12:39:29.861229 1 performance_metrics.go:329] Loaded 46 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-4.0-h-tiny-FP8-dynamic I0911 12:39:29.861263 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/phi-4 with 1 artifact(s) I0911 12:39:29.999856 1 performance_metrics.go:329] Loaded 55 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/phi-4 I0911 12:39:29.999893 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/phi-4-FP8-dynamic with 1 artifact(s) I0911 12:39:30.120791 1 performance_metrics.go:329] Loaded 54 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/phi-4-FP8-dynamic I0911 12:39:30.120825 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/phi-4-quantized.w4a16 with 1 artifact(s) I0911 12:39:30.295714 1 performance_metrics.go:329] Loaded 52 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/phi-4-quantized.w4a16 I0911 12:39:30.295751 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/phi-4-quantized.w8a8 with 1 artifact(s) I0911 12:39:30.429099 1 performance_metrics.go:329] Loaded 60 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/phi-4-quantized.w8a8 I0911 12:39:30.429135 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/sarvam-105b-FP8-dynamic with 1 artifact(s) I0911 12:39:30.594933 1 performance_metrics.go:329] Loaded 61 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/sarvam-105b-FP8-dynamic I0911 12:39:30.594993 1 loader.go:315] Loading model redhat_ai_validated_models:RedHatAI/sarvam-30b-FP8-dynamic with 1 artifact(s) I0911 12:39:30.595076 1 loader.go:453] redhat_ai_validated_models: loaded 89 models I0911 12:39:30.640021 1 loader.go:470] redhat_ai_validated_models: cleaned up 0 models I0911 12:39:30.846077 1 performance_metrics.go:329] Loaded 105 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/sarvam-30b-FP8-dynamic 2026/09/11 12:39:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49852 - 200 81B in 70.227µs 2026/09/11 12:39:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49882 - 200 81B in 28.537µs 2026/09/11 12:39:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54736 - 200 81B in 36.963µs W0911 12:39:45.411691 1 loader_base.go:100] Timeout waiting for inflight writes to complete I0911 12:39:47.393804 1 base.go:139] model plugin leader mode active 2026/09/11 12:39:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54762 - 200 81B in 38.227µs 2026/09/11 12:39:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39894 - 200 81B in 36.799µs 2026/09/11 12:39:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39900 - 200 81B in 42.809µs 2026/09/11 12:40:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47436 - 200 81B in 36.897µs 2026/09/11 12:40:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47458 - 200 81B in 36.096µs 2026/09/11 12:40:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36340 - 200 81B in 42.65µs 2026/09/11 12:40:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36358 - 200 81B in 37.363µs 2026/09/11 12:40:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54806 - 200 81B in 48.54µs 2026/09/11 12:40:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54810 - 200 81B in 39.728µs 2026/09/11 12:40:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44272 - 200 81B in 35.484µs 2026/09/11 12:40:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44286 - 200 81B in 45.744µs 2026/09/11 12:40:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52188 - 200 81B in 39.046µs 2026/09/11 12:40:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52206 - 200 81B in 35.807µs 2026/09/11 12:40:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36984 - 200 81B in 36.615µs 2026/09/11 12:40:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36992 - 200 81B in 42.212µs 2026/09/11 12:41:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40744 - 200 81B in 35.801µs 2026/09/11 12:41:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40752 - 200 81B in 36.079µs 2026/09/11 12:41:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55050 - 200 81B in 35.987µs 2026/09/11 12:41:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55060 - 200 81B in 44.218µs 2026/09/11 12:41:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38322 - 200 81B in 35.543µs 2026/09/11 12:41:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38334 - 200 81B in 36.177µs 2026/09/11 12:41:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48040 - 200 81B in 100.462µs 2026/09/11 12:41:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48058 - 200 81B in 43.324µs 2026/09/11 12:41:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44966 - 200 81B in 27.27µs 2026/09/11 12:41:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44992 - 200 81B in 38.286µs 2026/09/11 12:41:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36118 - 200 81B in 37.722µs 2026/09/11 12:41:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36124 - 200 81B in 39.157µs 2026/09/11 12:42:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55652 - 200 81B in 38.569µs 2026/09/11 12:42:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55658 - 200 81B in 38.72µs 2026/09/11 12:42:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37822 - 200 81B in 39.287µs 2026/09/11 12:42:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37844 - 200 81B in 43.044µs 2026/09/11 12:42:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54614 - 200 81B in 37.066µs 2026/09/11 12:42:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54636 - 200 81B in 42.243µs 2026/09/11 12:42:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56646 - 200 81B in 36.612µs 2026/09/11 12:42:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56658 - 200 81B in 37.68µs 2026/09/11 12:42:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60992 - 200 81B in 36.604µs 2026/09/11 12:42:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:32770 - 200 81B in 37.506µs 2026/09/11 12:42:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36262 - 200 81B in 37.509µs 2026/09/11 12:42:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36284 - 200 81B in 37.33µs 2026/09/11 12:43:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58610 - 200 81B in 43.929µs 2026/09/11 12:43:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58622 - 200 81B in 60.946µs 2026/09/11 12:43:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45052 - 200 81B in 36.948µs 2026/09/11 12:43:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45074 - 200 81B in 35.151µs 2026/09/11 12:43:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58182 - 200 81B in 38.992µs 2026/09/11 12:43:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58194 - 200 81B in 40.222µs 2026/09/11 12:43:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40958 - 200 81B in 44.855µs 2026/09/11 12:43:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40978 - 200 81B in 22.781µs 2026/09/11 12:43:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47714 - 200 81B in 26.085µs 2026/09/11 12:43:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47738 - 200 81B in 39.93µs 2026/09/11 12:43:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34296 - 200 81B in 48.984µs 2026/09/11 12:43:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34314 - 200 81B in 39.658µs 2026/09/11 12:44:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47526 - 200 81B in 37.237µs 2026/09/11 12:44:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47538 - 200 81B in 23.567µs 2026/09/11 12:44:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52224 - 200 81B in 49.767µs 2026/09/11 12:44:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52234 - 200 81B in 38.13µs 2026/09/11 12:44:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45058 - 200 81B in 68.424µs 2026/09/11 12:44:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45082 - 200 81B in 40.373µs 2026/09/11 12:44:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:43044 - 200 81B in 38.286µs 2026/09/11 12:44:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:43062 - 200 81B in 37.238µs 2026/09/11 12:44:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45924 - 200 81B in 38.449µs 2026/09/11 12:44:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45950 - 200 81B in 39.487µs 2026/09/11 12:44:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51584 - 200 81B in 36.04µs 2026/09/11 12:44:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51612 - 200 81B in 68.671µs 2026/09/11 12:45:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36880 - 200 81B in 54.727µs 2026/09/11 12:45:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36890 - 200 81B in 38.546µs 2026/09/11 12:45:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51492 - 200 81B in 39.207µs 2026/09/11 12:45:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51498 - 200 81B in 39.977µs 2026/09/11 12:45:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36898 - 200 81B in 70.005µs 2026/09/11 12:45:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36930 - 200 81B in 37.111µs 2026/09/11 12:45:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33700 - 200 81B in 37.056µs 2026/09/11 12:45:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33706 - 200 81B in 42.663µs 2026/09/11 12:45:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44912 - 200 81B in 39.736µs 2026/09/11 12:45:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44930 - 200 81B in 25.524µs 2026/09/11 12:45:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41668 - 200 81B in 38.633µs 2026/09/11 12:45:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41688 - 200 81B in 39.271µs 2026/09/11 12:46:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38008 - 200 81B in 45.961µs 2026/09/11 12:46:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38028 - 200 81B in 38.527µs 2026/09/11 12:46:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45904 - 200 81B in 61.09µs 2026/09/11 12:46:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45932 - 200 81B in 38.569µs 2026/09/11 12:46:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39180 - 200 81B in 48.846µs 2026/09/11 12:46:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39208 - 200 81B in 38.835µs 2026/09/11 12:46:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60254 - 200 81B in 36.466µs 2026/09/11 12:46:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60280 - 200 81B in 37.83µs 2026/09/11 12:46:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35498 - 200 81B in 38.196µs 2026/09/11 12:46:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35524 - 200 81B in 39.182µs 2026/09/11 12:46:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55928 - 200 81B in 37.13µs 2026/09/11 12:46:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55950 - 200 81B in 36.912µs 2026/09/11 12:47:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40630 - 200 81B in 37.103µs 2026/09/11 12:47:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40634 - 200 81B in 46.961µs 2026/09/11 12:47:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39166 - 200 81B in 38.86µs 2026/09/11 12:47:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39176 - 200 81B in 40.438µs 2026/09/11 12:47:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48270 - 200 81B in 50.571µs 2026/09/11 12:47:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48284 - 200 81B in 40.691µs 2026/09/11 12:47:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60504 - 200 81B in 38.239µs 2026/09/11 12:47:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60520 - 200 81B in 45.676µs 2026/09/11 12:47:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57416 - 200 81B in 42.784µs 2026/09/11 12:47:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57426 - 200 81B in 38.723µs 2026/09/11 12:47:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53006 - 200 81B in 60.392µs 2026/09/11 12:47:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53024 - 200 81B in 81.454µs 2026/09/11 12:48:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55680 - 200 81B in 38.15µs 2026/09/11 12:48:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55700 - 200 81B in 37.8µs 2026/09/11 12:48:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:46864 - 200 81B in 48.633µs 2026/09/11 12:48:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:46882 - 200 81B in 57.24µs 2026/09/11 12:48:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55806 - 200 81B in 39.401µs 2026/09/11 12:48:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55820 - 200 81B in 47.05µs 2026/09/11 12:48:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40018 - 200 81B in 38.044µs 2026/09/11 12:48:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40036 - 200 81B in 39.364µs 2026/09/11 12:48:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60280 - 200 81B in 41.778µs 2026/09/11 12:48:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60310 - 200 81B in 36.732µs 2026/09/11 12:48:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:50038 - 200 81B in 38.546µs 2026/09/11 12:48:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:50066 - 200 81B in 45.118µs 2026/09/11 12:49:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40782 - 200 81B in 37.961µs 2026/09/11 12:49:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40802 - 200 81B in 39.794µs 2026/09/11 12:49:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33760 - 200 81B in 41.832µs 2026/09/11 12:49:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33764 - 200 81B in 37.539µs 2026/09/11 12:49:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47518 - 200 81B in 47.392µs 2026/09/11 12:49:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47538 - 200 81B in 68.58µs 2026/09/11 12:49:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58224 - 200 81B in 45.407µs 2026/09/11 12:49:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58242 - 200 81B in 38.997µs 2026/09/11 12:49:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44804 - 200 81B in 47.933µs 2026/09/11 12:49:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44814 - 200 81B in 50.092µs 2026/09/11 12:49:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56266 - 200 81B in 22.706µs 2026/09/11 12:49:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56290 - 200 81B in 36.371µs 2026/09/11 12:50:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49772 - 200 81B in 42.203µs 2026/09/11 12:50:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49792 - 200 81B in 38.019µs 2026/09/11 12:50:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53302 - 200 81B in 38.189µs 2026/09/11 12:50:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53318 - 200 81B in 21.392µs 2026/09/11 12:50:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42728 - 200 81B in 43.622µs 2026/09/11 12:50:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42744 - 200 81B in 58.167µs 2026/09/11 12:50:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56662 - 200 81B in 61.518µs 2026/09/11 12:50:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56666 - 200 81B in 38.582µs 2026/09/11 12:50:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56508 - 200 81B in 50.01µs 2026/09/11 12:50:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56524 - 200 81B in 54.546µs 2026/09/11 12:50:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48584 - 200 81B in 37.999µs 2026/09/11 12:50:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48594 - 200 81B in 37.481µs 2026/09/11 12:51:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42528 - 200 81B in 40.35µs 2026/09/11 12:51:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42554 - 200 81B in 37.421µs 2026/09/11 12:51:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55084 - 200 81B in 40.262µs 2026/09/11 12:51:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55112 - 200 81B in 56.547µs 2026/09/11 12:51:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54476 - 200 81B in 42.378µs 2026/09/11 12:51:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54494 - 200 81B in 42.143µs 2026/09/11 12:51:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36918 - 200 81B in 36.167µs 2026/09/11 12:51:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36936 - 200 81B in 60.046µs 2026/09/11 12:51:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39732 - 200 81B in 24.907µs 2026/09/11 12:51:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39744 - 200 81B in 38.828µs 2026/09/11 12:51:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45956 - 200 81B in 21.838µs 2026/09/11 12:51:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45978 - 200 81B in 47.404µs 2026/09/11 12:52:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:32878 - 200 81B in 39.83µs 2026/09/11 12:52:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:32898 - 200 81B in 35.49µs 2026/09/11 12:52:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49818 - 200 81B in 22.04µs 2026/09/11 12:52:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49840 - 200 81B in 48.792µs 2026/09/11 12:52:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44654 - 200 81B in 37.795µs 2026/09/11 12:52:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44674 - 200 81B in 39.361µs 2026/09/11 12:52:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59930 - 200 81B in 37.512µs 2026/09/11 12:52:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59944 - 200 81B in 37.546µs 2026/09/11 12:52:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55034 - 200 81B in 39.497µs 2026/09/11 12:52:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55056 - 200 81B in 38.599µs 2026/09/11 12:52:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45338 - 200 81B in 43.947µs 2026/09/11 12:52:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45352 - 200 81B in 38.928µs 2026/09/11 12:53:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52910 - 200 81B in 52.352µs 2026/09/11 12:53:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52926 - 200 81B in 42.48µs 2026/09/11 12:53:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55522 - 200 81B in 48.525µs 2026/09/11 12:53:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55540 - 200 81B in 42.275µs 2026/09/11 12:53:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53348 - 200 81B in 37.323µs 2026/09/11 12:53:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53360 - 200 81B in 36.519µs 2026/09/11 12:53:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34700 - 200 81B in 36.084µs 2026/09/11 12:53:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34708 - 200 81B in 59.173µs 2026/09/11 12:53:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52772 - 200 81B in 62.53µs 2026/09/11 12:53:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52802 - 200 81B in 40.959µs 2026/09/11 12:53:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34934 - 200 81B in 27.355µs 2026/09/11 12:53:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34948 - 200 81B in 41.575µs 2026/09/11 12:54:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35010 - 200 81B in 36.801µs 2026/09/11 12:54:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35040 - 200 81B in 39.367µs 2026/09/11 12:54:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:50966 - 200 81B in 46.071µs 2026/09/11 12:54:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:50980 - 200 81B in 38.258µs 2026/09/11 12:54:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33622 - 200 81B in 61.825µs 2026/09/11 12:54:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33650 - 200 81B in 38.28µs 2026/09/11 12:54:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51768 - 200 81B in 38.727µs 2026/09/11 12:54:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51784 - 200 81B in 39.671µs 2026/09/11 12:54:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59362 - 200 81B in 43.401µs 2026/09/11 12:54:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59380 - 200 81B in 66.078µs 2026/09/11 12:54:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38008 - 200 81B in 37.943µs 2026/09/11 12:54:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38020 - 200 81B in 38.15µs 2026/09/11 12:55:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37728 - 200 81B in 41.182µs 2026/09/11 12:55:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37740 - 200 81B in 52.654µs 2026/09/11 12:55:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45168 - 200 81B in 38.572µs 2026/09/11 12:55:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45186 - 200 81B in 39.175µs 2026/09/11 12:55:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57686 - 200 81B in 55.256µs 2026/09/11 12:55:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57700 - 200 81B in 39.953µs 2026/09/11 12:55:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38034 - 200 81B in 40.459µs 2026/09/11 12:55:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38066 - 200 81B in 37.485µs 2026/09/11 12:55:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:32932 - 200 81B in 39.81µs 2026/09/11 12:55:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:32938 - 200 81B in 65.485µs 2026/09/11 12:55:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47400 - 200 81B in 36.079µs 2026/09/11 12:55:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47420 - 200 81B in 29.935µs 2026/09/11 12:56:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41868 - 200 81B in 69.558µs 2026/09/11 12:56:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41888 - 200 81B in 37.853µs 2026/09/11 12:56:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42662 - 200 81B in 51.147µs 2026/09/11 12:56:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42688 - 200 81B in 39.455µs 2026/09/11 12:56:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49044 - 200 81B in 39.973µs 2026/09/11 12:56:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49058 - 200 81B in 42.202µs 2026/09/11 12:56:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53534 - 200 81B in 38.755µs 2026/09/11 12:56:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53550 - 200 81B in 36.861µs 2026/09/11 12:56:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56782 - 200 81B in 38.359µs 2026/09/11 12:56:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56808 - 200 81B in 42.158µs 2026/09/11 12:56:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56616 - 200 81B in 38.14µs 2026/09/11 12:56:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56648 - 200 81B in 40.268µs 2026/09/11 12:57:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57600 - 200 81B in 37.751µs 2026/09/11 12:57:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57612 - 200 81B in 55.118µs 2026/09/11 12:57:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:46116 - 200 81B in 39.676µs 2026/09/11 12:57:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:46134 - 200 81B in 43.046µs 2026/09/11 12:57:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41566 - 200 81B in 80.586µs 2026/09/11 12:57:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41578 - 200 81B in 37.85µs 2026/09/11 12:57:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41046 - 200 81B in 38.952µs 2026/09/11 12:57:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41066 - 200 81B in 36.251µs 2026/09/11 12:57:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40104 - 200 81B in 32.887µs 2026/09/11 12:57:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40126 - 200 81B in 24.343µs 2026/09/11 12:57:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:43260 - 200 81B in 38.381µs 2026/09/11 12:57:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:43290 - 200 81B in 21.325µs 2026/09/11 12:58:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53458 - 200 81B in 39.407µs 2026/09/11 12:58:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53476 - 200 81B in 37.539µs 2026/09/11 12:58:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55678 - 200 81B in 38.039µs 2026/09/11 12:58:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55700 - 200 81B in 40.576µs 2026/09/11 12:58:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44780 - 200 81B in 37.449µs 2026/09/11 12:58:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44798 - 200 81B in 37.318µs 2026/09/11 12:58:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40400 - 200 81B in 38.082µs 2026/09/11 12:58:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40418 - 200 81B in 42.294µs 2026/09/11 12:58:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48054 - 200 81B in 49.262µs 2026/09/11 12:58:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48078 - 200 81B in 41.01µs 2026/09/11 12:58:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57280 - 200 81B in 37.939µs 2026/09/11 12:58:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57308 - 200 81B in 39.203µs 2026/09/11 12:59:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37958 - 200 81B in 37.465µs 2026/09/11 12:59:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37972 - 200 81B in 52.992µs 2026/09/11 12:59:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34696 - 200 81B in 39.673µs 2026/09/11 12:59:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34712 - 200 81B in 37.63µs 2026/09/11 12:59:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:46498 - 200 81B in 54.64µs 2026/09/11 12:59:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:46510 - 200 81B in 39.144µs 2026/09/11 12:59:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53056 - 200 81B in 37.453µs 2026/09/11 12:59:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53070 - 200 81B in 75.774µs 2026/09/11 12:59:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42570 - 200 81B in 23.957µs 2026/09/11 12:59:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42584 - 200 81B in 28.541µs 2026/09/11 12:59:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55110 - 200 81B in 37.369µs 2026/09/11 12:59:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55116 - 200 81B in 23.34µs 2026/09/11 13:00:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59282 - 200 81B in 39.753µs 2026/09/11 13:00:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59288 - 200 81B in 50.503µs 2026/09/11 13:00:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:50432 - 200 81B in 40.528µs 2026/09/11 13:00:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:50442 - 200 81B in 64.344µs 2026/09/11 13:00:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37346 - 200 81B in 64.464µs 2026/09/11 13:00:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37354 - 200 81B in 47.784µs 2026/09/11 13:00:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40008 - 200 81B in 49.607µs 2026/09/11 13:00:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40032 - 200 81B in 37.379µs 2026/09/11 13:00:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:43048 - 200 81B in 38.21µs 2026/09/11 13:00:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:43086 - 200 81B in 39.203µs 2026/09/11 13:00:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38308 - 200 81B in 57.875µs 2026/09/11 13:00:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38326 - 200 81B in 39.666µs 2026/09/11 13:01:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40276 - 200 81B in 38.945µs 2026/09/11 13:01:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40294 - 200 81B in 36.699µs 2026/09/11 13:01:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33058 - 200 81B in 52.099µs 2026/09/11 13:01:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33084 - 200 81B in 37.496µs 2026/09/11 13:01:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58580 - 200 81B in 51.732µs 2026/09/11 13:01:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58592 - 200 81B in 42.853µs 2026/09/11 13:01:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58112 - 200 81B in 38.934µs 2026/09/11 13:01:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58138 - 200 81B in 40.349µs 2026/09/11 13:01:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41064 - 200 81B in 28.236µs 2026/09/11 13:01:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41070 - 200 81B in 38.181µs 2026/09/11 13:01:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47922 - 200 81B in 41.627µs 2026/09/11 13:01:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47938 - 200 81B in 57.36µs 2026/09/11 13:02:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35090 - 200 81B in 39.677µs 2026/09/11 13:02:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35104 - 200 81B in 37.739µs 2026/09/11 13:02:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57782 - 200 81B in 47.325µs 2026/09/11 13:02:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57802 - 200 81B in 38.458µs 2026/09/11 13:02:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38356 - 200 81B in 27.345µs 2026/09/11 13:02:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38386 - 200 81B in 62.132µs 2026/09/11 13:02:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38740 - 200 81B in 39.213µs 2026/09/11 13:02:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38748 - 200 81B in 39.077µs 2026/09/11 13:02:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42846 - 200 81B in 21.79µs 2026/09/11 13:02:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42866 - 200 81B in 37.715µs 2026/09/11 13:02:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52562 - 200 81B in 47.655µs 2026/09/11 13:02:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52580 - 200 81B in 53.114µs 2026/09/11 13:03:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:46362 - 200 81B in 38.771µs 2026/09/11 13:03:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:46386 - 200 81B in 38.987µs 2026/09/11 13:03:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39566 - 200 81B in 74.596µs 2026/09/11 13:03:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39592 - 200 81B in 36.87µs 2026/09/11 13:03:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40636 - 200 81B in 38.199µs 2026/09/11 13:03:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40654 - 200 81B in 40.288µs 2026/09/11 13:03:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48380 - 200 81B in 38.763µs 2026/09/11 13:03:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48398 - 200 81B in 53.598µs 2026/09/11 13:03:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40378 - 200 81B in 53.882µs 2026/09/11 13:03:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40408 - 200 81B in 71.496µs 2026/09/11 13:03:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36674 - 200 81B in 40.234µs 2026/09/11 13:03:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36690 - 200 81B in 39.985µs 2026/09/11 13:04:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44360 - 200 81B in 37.317µs 2026/09/11 13:04:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44376 - 200 81B in 39.444µs 2026/09/11 13:04:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34160 - 200 81B in 62.028µs 2026/09/11 13:04:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34176 - 200 81B in 40.113µs 2026/09/11 13:04:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39634 - 200 81B in 47.96µs 2026/09/11 13:04:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39664 - 200 81B in 62.389µs 2026/09/11 13:04:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57540 - 200 81B in 37.7µs 2026/09/11 13:04:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57558 - 200 81B in 41.384µs 2026/09/11 13:04:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35024 - 200 81B in 40.429µs 2026/09/11 13:04:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35042 - 200 81B in 38.302µs 2026/09/11 13:04:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45050 - 200 81B in 68.789µs 2026/09/11 13:04:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45062 - 200 81B in 36.834µs 2026/09/11 13:05:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49598 - 200 81B in 64.432µs 2026/09/11 13:05:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49626 - 200 81B in 49.462µs 2026/09/11 13:05:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49482 - 200 81B in 38.346µs 2026/09/11 13:05:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49494 - 200 81B in 44.809µs 2026/09/11 13:05:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36582 - 200 81B in 46.78µs 2026/09/11 13:05:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36598 - 200 81B in 42.229µs 2026/09/11 13:05:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51064 - 200 81B in 36.538µs 2026/09/11 13:05:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51080 - 200 81B in 37.916µs 2026/09/11 13:05:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54420 - 200 81B in 44.284µs 2026/09/11 13:05:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54446 - 200 81B in 39.089µs 2026/09/11 13:05:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58392 - 200 81B in 39.172µs 2026/09/11 13:05:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58412 - 200 81B in 73.545µs 2026/09/11 13:06:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54884 - 200 81B in 38.309µs 2026/09/11 13:06:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54904 - 200 81B in 25.595µs 2026/09/11 13:06:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60300 - 200 81B in 38.112µs 2026/09/11 13:06:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60304 - 200 81B in 23.611µs 2026/09/11 13:06:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55890 - 200 81B in 41.975µs 2026/09/11 13:06:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55906 - 200 81B in 38.693µs 2026/09/11 13:06:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58864 - 200 81B in 40.17µs 2026/09/11 13:06:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58880 - 200 81B in 38.296µs 2026/09/11 13:06:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59128 - 200 81B in 43.654µs 2026/09/11 13:06:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59160 - 200 81B in 38.289µs 2026/09/11 13:06:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53316 - 200 81B in 37.131µs 2026/09/11 13:06:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53342 - 200 81B in 59.086µs 2026/09/11 13:07:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38246 - 200 81B in 37.607µs 2026/09/11 13:07:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:38270 - 200 81B in 45.818µs 2026/09/11 13:07:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58192 - 200 81B in 59.823µs 2026/09/11 13:07:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58222 - 200 81B in 37.145µs 2026/09/11 13:07:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48952 - 200 81B in 44.83µs 2026/09/11 13:07:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:48960 - 200 81B in 39.534µs 2026/09/11 13:07:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56054 - 200 81B in 39.797µs 2026/09/11 13:07:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:56076 - 200 81B in 61.064µs 2026/09/11 13:07:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41462 - 200 81B in 38.726µs 2026/09/11 13:07:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41482 - 200 81B in 23.677µs 2026/09/11 13:07:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45248 - 200 81B in 24.148µs 2026/09/11 13:07:54 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56242 - 200 420B in 4.028321ms 2026/09/11 13:07:55 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56242 - 200 420B in 1.323104ms 2026/09/11 13:07:57 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56242 - 200 420B in 1.692768ms 2026/09/11 13:07:57 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56408 - 200 420B in 1.135816ms 2026/09/11 13:07:57 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56408 - 200 420B in 1.244691ms 2026/09/11 13:07:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45276 - 200 81B in 37.309µs 2026/09/11 13:08:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53906 - 200 81B in 54.725µs 2026/09/11 13:08:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53936 - 200 81B in 40.634µs 2026/09/11 13:08:09 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56408 - 200 420B in 3.451172ms 2026/09/11 13:08:10 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56408 - 200 420B in 1.098771ms 2026/09/11 13:08:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45904 - 200 81B in 36.865µs 2026/09/11 13:08:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45920 - 200 81B in 44.542µs 2026/09/11 13:08:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57632 - 200 81B in 39.272µs 2026/09/11 13:08:29 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:56408 - 200 620B in 112.484µs 2026/09/11 13:08:29 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56242 - 200 420B in 3.799353ms 2026/09/11 13:08:29 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:56408 - 200 10876B in 5.715213ms 2026/09/11 13:08:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57656 - 200 81B in 87.213µs 2026/09/11 13:08:29 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:56242 - 200 620B in 92.899µs 2026/09/11 13:08:29 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56408 - 200 420B in 1.111512ms 2026/09/11 13:08:29 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:56242 - 200 10876B in 2.042674ms 2026/09/11 13:08:30 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:56242 - 200 47545B in 10.059404ms 2026/09/11 13:08:30 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:56408 - 200 126879B in 13.582943ms 2026/09/11 13:08:30 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:51520 - 200 119941B in 24.907332ms 2026/09/11 13:08:31 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fdiffusiongemma-26B-A4B-it-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58160 - 200 56B in 61.175386ms 2026/09/11 13:08:31 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56408 - 200 290332B in 105.176401ms 2026/09/11 13:08:31 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56242 - 200 324538B in 112.855681ms 2026/09/11 13:08:31 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58152 - 200 186447B in 113.909634ms 2026/09/11 13:08:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57806 - 200 81B in 28.684µs 2026/09/11 13:08:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57826 - 200 81B in 44.384µs 2026/09/11 13:08:43 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:56408 - 200 620B in 110.281µs 2026/09/11 13:08:43 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56408 - 200 420B in 1.92022ms 2026/09/11 13:08:43 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:58160 - 200 10876B in 4.969948ms 2026/09/11 13:08:43 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:56408 - 200 620B in 62.834µs 2026/09/11 13:08:43 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:58160 - 200 420B in 1.319165ms 2026/09/11 13:08:43 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:35382 - 200 10876B in 3.923349ms 2026/09/11 13:08:44 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:58160 - 200 47545B in 8.068954ms 2026/09/11 13:08:44 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:56408 - 200 126879B in 10.260028ms 2026/09/11 13:08:44 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:35390 - 200 119941B in 24.424493ms 2026/09/11 13:08:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36044 - 200 81B in 51.712µs 2026/09/11 13:08:44 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fdiffusiongemma-26B-A4B-it-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:35406 - 200 56B in 23.158972ms 2026/09/11 13:08:44 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56408 - 200 186447B in 93.654561ms 2026/09/11 13:08:44 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58160 - 200 290332B in 106.204263ms 2026/09/11 13:08:44 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:35406 - 200 324538B in 85.29696ms 2026/09/11 13:08:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36060 - 200 81B in 54.86µs 2026/09/11 13:08:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37348 - 200 81B in 22.099µs 2026/09/11 13:08:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37362 - 200 81B in 41.367µs 2026/09/11 13:09:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58684 - 200 81B in 54.673µs 2026/09/11 13:09:06 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:58160 - 200 620B in 102.412µs 2026/09/11 13:09:06 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56408 - 200 420B in 3.19889ms 2026/09/11 13:09:06 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:58160 - 200 10876B in 4.317883ms 2026/09/11 13:09:06 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:56408 - 200 620B in 65.784µs 2026/09/11 13:09:06 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:58160 - 200 420B in 2.182205ms 2026/09/11 13:09:06 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:56408 - 200 10876B in 4.06069ms 2026/09/11 13:09:06 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:56408 - 200 47545B in 8.752895ms 2026/09/11 13:09:06 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:58160 - 200 126879B in 13.04945ms 2026/09/11 13:09:06 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:56408 - 200 119941B in 9.005118ms 2026/09/11 13:09:07 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fdiffusiongemma-26B-A4B-it-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58160 - 200 56B in 11.281832ms 2026/09/11 13:09:07 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56408 - 200 290332B in 74.591046ms 2026/09/11 13:09:07 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53018 - 200 186447B in 146.886691ms 2026/09/11 13:09:07 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53006 - 200 324538B in 184.181219ms 2026/09/11 13:09:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58702 - 200 81B in 25.177µs 2026/09/11 13:09:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40226 - 200 81B in 23.94µs 2026/09/11 13:09:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40242 - 200 81B in 19.227µs 2026/09/11 13:09:19 "GET http://rh-ai.apps.e8639f3e16b6ea55ad21.prod.konflux-ocp-ci.dev/api/model_catalog/v1alpha1/models?filterQuery=artifacts.use_case.string_value%3D%27chatbot%27+AND+artifacts.ttft_p90.double_value+%3C%3D+177772.9208469391&latencyProperty=ttft_p90&orderBy=RECOMMENDED&pageSize=10&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56408 - 200 294459B in 2.043235541s 2026/09/11 13:09:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59028 - 200 81B in 38.15µs 2026/09/11 13:09:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59042 - 200 81B in 37.751µs 2026/09/11 13:09:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39504 - 200 81B in 38.367µs 2026/09/11 13:09:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39514 - 200 81B in 22.43µs 2026/09/11 13:09:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44360 - 200 81B in 39.019µs 2026/09/11 13:09:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44370 - 200 81B in 23.858µs 2026/09/11 13:09:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60578 - 200 81B in 55.88µs 2026/09/11 13:09:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60604 - 200 81B in 42.194µs 2026/09/11 13:10:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35912 - 200 81B in 47.043µs 2026/09/11 13:10:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35930 - 200 81B in 46.692µs 2026/09/11 13:10:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57030 - 200 81B in 41.115µs 2026/09/11 13:10:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57050 - 200 81B in 37.919µs 2026/09/11 13:10:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33390 - 200 81B in 38.171µs 2026/09/11 13:10:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:33414 - 200 81B in 40.719µs 2026/09/11 13:10:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58178 - 200 81B in 37.685µs 2026/09/11 13:10:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58182 - 200 81B in 40.323µs 2026/09/11 13:10:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58666 - 200 81B in 39.009µs 2026/09/11 13:10:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:58678 - 200 81B in 37.951µs 2026/09/11 13:10:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59088 - 200 81B in 36.62µs 2026/09/11 13:10:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59100 - 200 81B in 38.897µs 2026/09/11 13:11:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59092 - 200 81B in 37.642µs 2026/09/11 13:11:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59110 - 200 81B in 43.151µs 2026/09/11 13:11:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35478 - 200 81B in 38.511µs 2026/09/11 13:11:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35488 - 200 81B in 42.199µs 2026/09/11 13:11:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40838 - 200 81B in 40.618µs 2026/09/11 13:11:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40854 - 200 81B in 55.143µs 2026/09/11 13:11:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47794 - 200 81B in 39.254µs 2026/09/11 13:11:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47808 - 200 81B in 37.934µs 2026/09/11 13:11:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60864 - 200 81B in 39.806µs 2026/09/11 13:11:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60880 - 200 81B in 39.412µs 2026/09/11 13:11:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47226 - 200 81B in 37.095µs 2026/09/11 13:11:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47254 - 200 81B in 44.82µs 2026/09/11 13:12:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:43990 - 200 81B in 38.228µs 2026/09/11 13:12:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44004 - 200 81B in 36.51µs 2026/09/11 13:12:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41596 - 200 81B in 22.753µs 2026/09/11 13:12:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41618 - 200 81B in 38.447µs 2026/09/11 13:12:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36384 - 200 81B in 39.287µs 2026/09/11 13:12:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:36392 - 200 81B in 38.468µs 2026/09/11 13:12:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51410 - 200 81B in 34.896µs 2026/09/11 13:12:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51426 - 200 81B in 36.327µs 2026/09/11 13:12:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54170 - 200 81B in 41.671µs 2026/09/11 13:12:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54190 - 200 81B in 40.132µs 2026/09/11 13:12:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34966 - 200 81B in 36.883µs 2026/09/11 13:12:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34986 - 200 81B in 38.718µs 2026/09/11 13:13:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59594 - 200 81B in 38.999µs 2026/09/11 13:13:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59618 - 200 81B in 38.602µs 2026/09/11 13:13:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52644 - 200 81B in 37.022µs 2026/09/11 13:13:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:52662 - 200 81B in 64.19µs 2026/09/11 13:13:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35544 - 200 81B in 56.636µs 2026/09/11 13:13:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35560 - 200 81B in 25.238µs 2026/09/11 13:13:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60646 - 200 81B in 46.803µs 2026/09/11 13:13:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60662 - 200 81B in 38.787µs 2026/09/11 13:13:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40298 - 200 81B in 45.123µs 2026/09/11 13:13:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:40312 - 200 81B in 22.142µs 2026/09/11 13:13:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:32892 - 200 81B in 70.144µs 2026/09/11 13:13:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:32914 - 200 81B in 38.931µs 2026/09/11 13:14:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34228 - 200 81B in 38.909µs 2026/09/11 13:14:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34242 - 200 81B in 42.91µs 2026/09/11 13:14:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47976 - 200 81B in 37.92µs 2026/09/11 13:14:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47994 - 200 81B in 38.583µs 2026/09/11 13:14:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39436 - 200 81B in 36.908µs 2026/09/11 13:14:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:39462 - 200 81B in 37.378µs 2026/09/11 13:14:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45416 - 200 81B in 37.841µs 2026/09/11 13:14:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45434 - 200 81B in 40.448µs 2026/09/11 13:14:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54902 - 200 81B in 46.535µs 2026/09/11 13:14:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:54918 - 200 81B in 41.312µs 2026/09/11 13:14:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42356 - 200 81B in 38.837µs 2026/09/11 13:14:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42382 - 200 81B in 36.297µs 2026/09/11 13:15:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57402 - 200 81B in 61.585µs 2026/09/11 13:15:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:57410 - 200 81B in 40.037µs 2026/09/11 13:15:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53164 - 200 81B in 68.499µs 2026/09/11 13:15:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:53182 - 200 81B in 38.611µs 2026/09/11 13:15:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41798 - 200 81B in 160.119µs 2026/09/11 13:15:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41814 - 200 81B in 23.44µs 2026/09/11 13:15:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41458 - 200 81B in 39.62µs 2026/09/11 13:15:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41470 - 200 81B in 39.85µs 2026/09/11 13:15:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59630 - 200 81B in 53.897µs 2026/09/11 13:15:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:59648 - 200 81B in 46.915µs 2026/09/11 13:15:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55372 - 200 81B in 40.035µs 2026/09/11 13:15:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:55390 - 200 81B in 63.354µs 2026/09/11 13:16:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42636 - 200 81B in 37.624µs 2026/09/11 13:16:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:42648 - 200 81B in 54.597µs 2026/09/11 13:16:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51564 - 200 81B in 47.807µs 2026/09/11 13:16:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:51588 - 200 81B in 38.323µs 2026/09/11 13:16:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41200 - 200 81B in 47.742µs 2026/09/11 13:16:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:41208 - 200 81B in 23.047µs 2026/09/11 13:16:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45384 - 200 81B in 45.179µs 2026/09/11 13:16:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45396 - 200 81B in 37.424µs 2026/09/11 13:16:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47258 - 200 81B in 66.861µs 2026/09/11 13:16:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:47278 - 200 81B in 55.624µs 2026/09/11 13:16:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35488 - 200 81B in 37.745µs 2026/09/11 13:16:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35504 - 200 81B in 37.833µs 2026/09/11 13:17:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37912 - 200 81B in 55.096µs 2026/09/11 13:17:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37930 - 200 81B in 47.754µs 2026/09/11 13:17:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45824 - 200 81B in 47.02µs 2026/09/11 13:17:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:45848 - 200 81B in 43.768µs 2026/09/11 13:17:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34988 - 200 81B in 48.016µs 2026/09/11 13:17:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:35012 - 200 81B in 68.493µs 2026/09/11 13:17:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44410 - 200 81B in 42.042µs 2026/09/11 13:17:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44432 - 200 81B in 38.7µs 2026/09/11 13:17:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60792 - 200 81B in 45.429µs 2026/09/11 13:17:49 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:60802 - 200 81B in 52.649µs 2026/09/11 13:17:54 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34906 - 200 81B in 24.65µs 2026/09/11 13:17:59 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:34928 - 200 81B in 36.599µs 2026/09/11 13:18:04 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44266 - 200 81B in 36.707µs 2026/09/11 13:18:09 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:44278 - 200 81B in 38.728µs 2026/09/11 13:18:14 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37790 - 200 81B in 60.281µs 2026/09/11 13:18:19 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37804 - 200 81B in 37.216µs 2026/09/11 13:18:24 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49360 - 200 81B in 51.807µs 2026/09/11 13:18:29 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:49382 - 200 81B in 38.968µs 2026/09/11 13:18:34 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:43834 - 200 81B in 36.859µs 2026/09/11 13:18:39 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:43864 - 200 81B in 36.984µs 2026/09/11 13:18:44 "GET http://10.132.0.50:8080/readyz HTTP/1.1" from 10.132.0.2:37086 - 200 81B in 37.791µs W0911 12:38:52.121120 1 deprecated.go:66] ==== Removed Flag Warning ====================== logtostderr is removed in the k8s upstream and has no effect any more. =============================================== I0911 12:38:52.121187 1 kube-rbac-proxy.go:525] Reading config file: /etc/kube-rbac-proxy/config-file.yaml I0911 12:38:52.121870 1 kube-rbac-proxy.go:245] Valid token audiences: I0911 12:38:52.122761 1 kube-rbac-proxy.go:368] Reading certificate files I0911 12:38:52.123156 1 kube-rbac-proxy.go:404] Starting TCP socket on 0.0.0.0:8443 I0911 12:38:52.123183 1 kube-rbac-proxy.go:441] Starting TCP socket on 0.0.0.0:8888 I0911 12:38:52.123600 1 kube-rbac-proxy.go:411] Listening securely on 0.0.0.0:8443 I0911 12:38:52.123601 1 kube-rbac-proxy.go:448] Listening securely on 0.0.0.0:8888 for proxy endpoints ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-5978ccb457-j9cfs.log ==================== The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "en_US.utf8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english". Data page checksums are disabled. fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok creating subdirectories ... ok selecting dynamic shared memory implementation ... posix selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting default time zone ... UTC creating configuration files ... ok running bootstrap script ... ok performing post-bootstrap initialization ... ok syncing data to disk ... ok Success. You can now start the database server using: pg_ctl -D /var/lib/pgsql/data/userdata -l logfile start initdb: warning: enabling "trust" authentication for local connections initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. waiting for server to start....2026-09-11 12:39:00.781 UTC [30] LOG: redirecting log output to logging collector process 2026-09-11 12:39:00.781 UTC [30] HINT: Future log output will appear in directory "log". done server started /var/run/postgresql:5432 - accepting connections => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ... ALTER ROLE waiting for server to shut down.... done server stopped Starting server... 2026-09-11 12:39:01.061 UTC [1] LOG: redirecting log output to logging collector process 2026-09-11 12:39:01.061 UTC [1] HINT: Future log output will appear in directory "log".