==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== 88m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 124m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 124m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 88m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 88m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m25s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m19s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m16s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m16s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m19s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m25s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 160m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 88m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 160m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 124m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 124m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 160m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 160m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m14s Warning TooManyMissedTimes cronjob/maas-api-key-cleanup too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew 33s Warning FailedMount pod/maas-api-e2e-aigw-cf933f2a-5f6d987c7b-kh96d MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-aigw-cf933f2a-serving-cert" not found {"level":"error","ts":"2026-07-12T08:53:55Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:54:37Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:55:16Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:56:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:56:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:57:37Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:58:34Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:59:11Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:00:03Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:01:03Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:01:37Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:02:32Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:03:28Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:04:06Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:05:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:05:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:06:38Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:07:16Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:08:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:08:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:09:32Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:10:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:10:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:11:28Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:12:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:12:44Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:13:39Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:14:31Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:15:17Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:15:57Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:16:44Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:17:31Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-07-12T09:18:09Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.19 Kubernetes Version: v1.34.8 --- Namespaces --- namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.1 Authorino Operator 1.4.1 authorino-operator.v1.3.2 Succeeded cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Succeeded custom-metrics-autoscaler.v2.19.0-1 Custom Metrics Autoscaler 2.19.0-1 custom-metrics-autoscaler.v2.18.1-2 Succeeded dns-operator.v1.4.0 DNS Operator 1.4.0 dns-operator.v1.3.1 Succeeded kueue-operator.v1.3.1 Red Hat build of Kueue 1.3.1 Succeeded limitador-operator.v1.4.0 Limitador Operator 1.4.0 limitador-operator.v1.3.1 Succeeded rhcl-operator.v1.4.1 Red Hat Connectivity Link 1.4.1 rhcl-operator.v1.3.5 Succeeded rhods-operator.3.5.0-ea.2 Red Hat OpenShift AI 3.5.0-ea.2 Succeeded servicemeshoperator3.v3.2.0 Red Hat OpenShift Service Mesh 3 3.2.0 servicemeshoperator3.v3.1.10 Pending servicemeshoperator3.v3.4.0 Red Hat OpenShift Service Mesh 3 3.4.0 servicemeshoperator3.v3.2.0 Pending --- CSVs not in Succeeded phase --- servicemeshoperator3.v3.2.0: phase=Pending reason=Copied message=The operator is running in openshift-operators but is managing this namespace servicemeshoperator3.v3.4.0: phase=Pending reason=Copied message=The operator is running in openshift-operators but is managing this namespace --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aigateway: managementState=n/a aipipelines: managementState=Managed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Unmanaged llamastackoperator: managementState=n/a mlflowoperator: managementState=Managed modelregistry: managementState=Managed ogx: managementState=Removed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Managed trustyai: managementState=Managed workbenches: managementState=Managed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= AIPipelinesReady: status=True reason= message= DashboardReady: status=True reason= message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=True reason= message= LLM-D-WVADependencies: status=True reason= message= MLflowOperatorReady: status=True reason= message= MaaSPrerequisitesAvailable: status=True reason=MaaSPrerequisitesMet message=MaaS gateway annotations and Authorino TLS are correctly configured ModelControllerReady: status=True reason= message= ModelRegistryReady: status=True reason= message= ModelsAsServiceReady: status=True reason=Reconciled message=MaaS platform manifests applied and maas-api deployment is available OGXReady: status=False reason=Removed message=Component ManagementState is set to Removed RayReady: status=True reason= message= SparkOperatorReady: status=True reason= message= TrainerReady: status=True reason= message= TrainingOperatorReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason= message= --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 2d3h Ready 2026-07-10T06:04:24Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in DSCI CR OpenTelemetryCollectorAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Metrics not configured in DSCI CR Traces not configured in DSCI CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in DSCI CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR PersesAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR --- Component CRs (Ready) --- NAME READY REASON URL dashboard.components.platform.opendatahub.io/default-dashboard True NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON kueue.components.platform.opendatahub.io/default-kueue True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True NAME READY REASON URI modelcontroller.components.platform.opendatahub.io/default-modelcontroller True NAME READY REASON modelregistry.components.platform.opendatahub.io/default-modelregistry True NAME READY REASON modelsasservice.components.platform.opendatahub.io/default-modelsasservice True NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True NAME READY REASON trainer.components.platform.opendatahub.io/default-trainer True NAME READY REASON trainingoperator.components.platform.opendatahub.io/default-trainingoperator True NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON workbenches.components.platform.opendatahub.io/default-workbenches True --- Deployments not fully ready (ODH/RHOAI namespaces) --- (all deployments report ready replicas == desired) --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- (all pods Running or Succeeded) --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 88m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 124m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 124m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 88m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 88m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m25s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m19s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m16s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m16s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m19s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m25s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 160m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 39m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 88m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 160m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 111m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 124m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 124m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 147m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 160m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 160m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m14s Warning TooManyMissedTimes cronjob/maas-api-key-cleanup too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew 33s Warning FailedMount pod/maas-api-e2e-aigw-cf933f2a-5f6d987c7b-kh96d MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-aigw-cf933f2a-serving-cert" not found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- (no events) --- namespace/rhoai-model-registries --- (no events) --- namespace/rhods-notebooks --- (no events) --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-07-12T08:53:29Z) --- redhat-ods-operator/rhods-operator-6d59b54db7-vnrh2 --- {"level":"error","ts":"2026-07-12T08:53:55Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:54:37Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:55:16Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:56:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:56:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:57:37Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:58:34Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:59:11Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:00:03Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:01:03Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:01:37Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:02:32Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:03:28Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:04:06Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:05:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:05:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:06:38Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:07:16Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:08:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:08:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:09:32Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:10:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:10:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:11:28Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:12:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:12:44Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:13:39Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:14:31Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:15:17Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:15:57Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:16:44Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:17:31Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-6d59b54db7-qs29r.log ==================== ==================== OPERATOR POD LOG: rhods-operator-6d59b54db7-rgvz5.log ==================== ==================== OPERATOR POD LOG: rhods-operator-6d59b54db7-vnrh2.log ==================== {"level":"error","ts":"2026-07-12T08:53:55Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:54:37Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T08:54:55Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T08:55:16Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T08:56:00Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T08:56:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T08:56:21Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:21Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"19c40442-6ab1-46bc-a83c-e984b3f476a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:22Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"5a50ccc5-ebcd-4daa-b36c-ab6292d80b32","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"5a50ccc5-ebcd-4daa-b36c-ab6292d80b32","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b6444a40-72e5-40a4-9205-d2780e563702","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:22Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e77b609c-8737-4635-8ec5-4605d034714e","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:22Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e77b609c-8737-4635-8ec5-4605d034714e","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:23Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14aa724c-96e5-4aa1-891b-c0fc8db811cc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:24Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"d22850a7-2e5c-4cab-a5f0-970c38fa8434","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:24Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"d22850a7-2e5c-4cab-a5f0-970c38fa8434","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.initializeModules"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules","msg":"module CR gone, cleaning up operator resources","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","module":"aigateway"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"076d67fb-0b35-4d9a-8b24-44cd28cbb719","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/mlflowoperator.initialize"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/mlflowoperator.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.initialize"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.argoWorkflowsControllersOptions"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"af89842a-6024-452c-80c4-873c6a7f178c"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"af89842a-6024-452c-80c4-873c6a7f178c"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"af89842a-6024-452c-80c4-873c6a7f178c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"af89842a-6024-452c-80c4-873c6a7f178c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"af89842a-6024-452c-80c4-873c6a7f178c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"af89842a-6024-452c-80c4-873c6a7f178c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.initialize"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.migrateDeploymentSelector"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.initialize"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.configureDependencies"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/sanitycheck.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"reconcile","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"apply","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.initialize"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.customizeManifests"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.configureDependencies"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:35Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-12T08:56:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"reconcile","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"61db1de4-e273-419f-a836-6a259df391b9"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"apply","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"61db1de4-e273-419f-a836-6a259df391b9"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"61db1de4-e273-419f-a836-6a259df391b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainingoperator.initialize"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"61db1de4-e273-419f-a836-6a259df391b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"61db1de4-e273-419f-a836-6a259df391b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"61db1de4-e273-419f-a836-6a259df391b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"reconcile","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"apply","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ray.initialize"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.initialize"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.createConfigMap"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"1a3a165d-ee37-4122-b267-6e6161add86a"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"1a3a165d-ee37-4122-b267-6e6161add86a"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"1a3a165d-ee37-4122-b267-6e6161add86a","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"1a3a165d-ee37-4122-b267-6e6161add86a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"1a3a165d-ee37-4122-b267-6e6161add86a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"1a3a165d-ee37-4122-b267-6e6161add86a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"reconcile","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"apply","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.initialize"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.deployObservabilityManifests"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:36Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a7366d4f-6e27-41d5-9315-88db87d6fc61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"264cbb8c-bda6-40dc-a032-33426ba641f6"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"264cbb8c-bda6-40dc-a032-33426ba641f6"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"264cbb8c-bda6-40dc-a032-33426ba641f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"264cbb8c-bda6-40dc-a032-33426ba641f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"0f46bd31-875c-4cce-a01d-ad8ef4f59a9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-12T08:56:37Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:37Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.configureDependencies"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:38Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2f2f1143-d90f-435c-a5a1-020b9bc55c79","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2f2f1143-d90f-435c-a5a1-020b9bc55c79","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b42aea08-899a-4a35-893b-abce55b18274","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:39Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"delete","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9","gvk":"components.platform.opendatahub.io/v1alpha1, Kind=OGX","ns":"","name":"default-ogx"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"bf62f5da-d43b-4488-a4ed-66f5ce5dc761"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"a02b5fa8-829c-492e-a03b-448b23d76817"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"f4556cf0-2854-463f-b9c5-b6c54a0c4a72"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"92604753-5334-46f9-ad90-c8fc96024dc1"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"e157ad86-96d5-49ce-b0f7-544a802c91f8"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"369b28f3-0893-484d-a832-5e00a85c0d62"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"02f8cfbc-3880-45cd-8d44-9cfad89edfb6"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"58e7784a-6d7b-4983-8621-20ff3ee20088"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"9b49ae24-2b34-4245-824d-f26ae0db3ad2"} {"level":"info","ts":"2026-07-12T08:56:39Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"63536d51-c39f-4c6c-bb92-917f86ea7860"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"26baffcf-4a7a-4838-8f9c-f18060d1eb8e"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"16eb8594-af2f-4d82-b0ec-75b88fba5afa"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"ef7532ff-4d02-473e-b8e6-df84b1856ff9"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-12T08:56:40Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"67258698-2944-4116-a7a3-ce2443abee99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c98f24c0-2d87-4524-bfe5-bf10f6000652","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c98f24c0-2d87-4524-bfe5-bf10f6000652","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1e830d79-84fc-4b94-b5db-403e5b7de19e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.updateStatus"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"74b268b7-c2e4-445f-a6a2-d7566f5c712f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"7e36dcac-5046-4ca3-a883-914903ad770e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0ed73420-4cc5-4deb-8261-3c6fbe680cd9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"88f4f34d-ff95-4096-8cde-56a2215a06f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-12T08:56:40Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bab1c448-8777-4968-a1f9-0696e9c24384"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bab1c448-8777-4968-a1f9-0696e9c24384"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bab1c448-8777-4968-a1f9-0696e9c24384","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bab1c448-8777-4968-a1f9-0696e9c24384","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bab1c448-8777-4968-a1f9-0696e9c24384","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bab1c448-8777-4968-a1f9-0696e9c24384","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bab1c448-8777-4968-a1f9-0696e9c24384","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"1d677f82-7e2b-48cd-9eaf-e71063e8c6a2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"45ed6f90-df68-4d0e-b9e9-e5bcf32d1918","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"45ed6f90-df68-4d0e-b9e9-e5bcf32d1918","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bab1c448-8777-4968-a1f9-0696e9c24384","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"bf60e86b-d4c3-417d-b085-2c3eb4b4209f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"244963f0-555f-4e73-ad5d-3cdac933264c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"df312928-605f-4687-8752-46b2d0d9bbf5"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"af89842a-6024-452c-80c4-873c6a7f178c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"885ea9ca-e27d-4edb-8491-2732def01613"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"264cbb8c-bda6-40dc-a032-33426ba641f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"264cbb8c-bda6-40dc-a032-33426ba641f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"264cbb8c-bda6-40dc-a032-33426ba641f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"264cbb8c-bda6-40dc-a032-33426ba641f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"af89842a-6024-452c-80c4-873c6a7f178c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"af89842a-6024-452c-80c4-873c6a7f178c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:41Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"e4902acc-8481-465d-9814-72929a60c322","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"1a3a165d-ee37-4122-b267-6e6161add86a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"1a3a165d-ee37-4122-b267-6e6161add86a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"1a3a165d-ee37-4122-b267-6e6161add86a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"bab1c448-8777-4968-a1f9-0696e9c24384","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7622b258-64f8-44ff-8b3e-0681198d8b1c"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7622b258-64f8-44ff-8b3e-0681198d8b1c"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7622b258-64f8-44ff-8b3e-0681198d8b1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7622b258-64f8-44ff-8b3e-0681198d8b1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7622b258-64f8-44ff-8b3e-0681198d8b1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7622b258-64f8-44ff-8b3e-0681198d8b1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7622b258-64f8-44ff-8b3e-0681198d8b1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"22807322-83e4-4278-8bbb-eafdb70d2297","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7622b258-64f8-44ff-8b3e-0681198d8b1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"61db1de4-e273-419f-a836-6a259df391b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"61db1de4-e273-419f-a836-6a259df391b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"61db1de4-e273-419f-a836-6a259df391b9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a2b78d96-e578-421a-bdcb-77220f7f926f","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a2b78d96-e578-421a-bdcb-77220f7f926f","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"92519579-7fac-401a-83c0-98409d17443f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"7622b258-64f8-44ff-8b3e-0681198d8b1c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b67cdeb6-7eac-4bf3-a113-4c0059ff526d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/imagestreams.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.updateStatus"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"34bb4ca2-50cf-4210-8b4d-245ea5a8ba74","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"34bb4ca2-50cf-4210-8b4d-245ea5a8ba74","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:43Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"d2f902c2-7c35-4ceb-aae3-cadf3836d28d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.reconcileHardwareProfiles"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.updateStatus"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"929c5101-4478-4119-afa7-307946c03f07","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-12T08:56:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:44Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cf0675b1-1663-4d68-9be0-28ed04f6b07a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"087c9b32-009a-4e01-9ac5-2f5cffdc526f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"11b7e39f-0c01-4e90-b47a-370e844115f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:45Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ca798b81-c1f3-405d-a6f7-f10f6f432541","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ca798b81-c1f3-405d-a6f7-f10f6f432541","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b933d8e6-05fc-43d7-a0b4-20f64d683088","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9601e82c-f06e-4004-b8bd-912dbfcd69ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:46Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"d21b10ff-c06e-4710-83af-b9218df501cf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:46Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a772b3f1-f774-4721-8752-3c56dc4dfdfa","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:46Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a772b3f1-f774-4721-8752-3c56dc4dfdfa","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5298253a-b654-4ebe-9bec-c175f669df2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:48Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"33c42eb6-f376-4480-a42f-8566fee4595e","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"33c42eb6-f376-4480-a42f-8566fee4595e","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"14d081b4-4665-42da-abfd-535fd000df36","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:56:50Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-12T08:56:50Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:50Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:56:50Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"ffe6be94-88c0-4a51-bcd2-2133b07cab92","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-12T08:56:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ca7cfba3-aad8-489c-9d9c-40f9080de84c"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ca7cfba3-aad8-489c-9d9c-40f9080de84c"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ca7cfba3-aad8-489c-9d9c-40f9080de84c","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ca7cfba3-aad8-489c-9d9c-40f9080de84c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ca7cfba3-aad8-489c-9d9c-40f9080de84c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ca7cfba3-aad8-489c-9d9c-40f9080de84c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.initialize"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.customizeManifests"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.configureDependencies"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"08dd7d55-8d78-4d63-854b-8068bdda5a3a"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"08dd7d55-8d78-4d63-854b-8068bdda5a3a"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"08dd7d55-8d78-4d63-854b-8068bdda5a3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.initialize"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.createConfigMap"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"08dd7d55-8d78-4d63-854b-8068bdda5a3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.initialize"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.migrateDeploymentSelector"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"04b37357-0164-4c83-95af-8dc7814d19d5"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"04b37357-0164-4c83-95af-8dc7814d19d5"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"04b37357-0164-4c83-95af-8dc7814d19d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"04b37357-0164-4c83-95af-8dc7814d19d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"04b37357-0164-4c83-95af-8dc7814d19d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"04b37357-0164-4c83-95af-8dc7814d19d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.initialize"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.configureDependencies"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.initialize"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.deployObservabilityManifests"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/sanitycheck.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"reconcile","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"7963e494-93bd-4e78-b060-b96333deba2e"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"apply","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"7963e494-93bd-4e78-b060-b96333deba2e"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"7963e494-93bd-4e78-b060-b96333deba2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainingoperator.initialize"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"7963e494-93bd-4e78-b060-b96333deba2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"7963e494-93bd-4e78-b060-b96333deba2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"7963e494-93bd-4e78-b060-b96333deba2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.configureDependencies"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ray.initialize"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:22Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"reconcile","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"apply","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.initialize"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.argoWorkflowsControllersOptions"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-12T08:57:23Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"reconcile","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"apply","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/mlflowoperator.initialize"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/mlflowoperator.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ca7cfba3-aad8-489c-9d9c-40f9080de84c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ca7cfba3-aad8-489c-9d9c-40f9080de84c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"ca7cfba3-aad8-489c-9d9c-40f9080de84c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.updateStatus"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e671c3f2-ff00-4688-9e7e-b1d7c1b962f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"6f1ccec3-71af-4abb-8b79-723c828c4951","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"13f14f27-4f3a-4d02-a535-bcae645acd91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"04b37357-0164-4c83-95af-8dc7814d19d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"04b37357-0164-4c83-95af-8dc7814d19d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"04b37357-0164-4c83-95af-8dc7814d19d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"7963e494-93bd-4e78-b060-b96333deba2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"7963e494-93bd-4e78-b060-b96333deba2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"7963e494-93bd-4e78-b060-b96333deba2e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"df847d27-7cb9-4bb8-839f-5bfaff58c045","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"036a6721-7921-489e-8880-0efe9dd62839","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2dc26ede-d285-4cb8-a141-cf35939fe610"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2dc26ede-d285-4cb8-a141-cf35939fe610"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2dc26ede-d285-4cb8-a141-cf35939fe610","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2dc26ede-d285-4cb8-a141-cf35939fe610","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2dc26ede-d285-4cb8-a141-cf35939fe610","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2dc26ede-d285-4cb8-a141-cf35939fe610","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ba6f3bc8-3e5b-438c-9bcd-5c42c35297ae"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ba6f3bc8-3e5b-438c-9bcd-5c42c35297ae"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ba6f3bc8-3e5b-438c-9bcd-5c42c35297ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ba6f3bc8-3e5b-438c-9bcd-5c42c35297ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ba6f3bc8-3e5b-438c-9bcd-5c42c35297ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"reconcile","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"apply","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.checkPreConditions"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.initialize"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/datasciencepipelines.argoWorkflowsControllersOptions"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2dc26ede-d285-4cb8-a141-cf35939fe610","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2dc26ede-d285-4cb8-a141-cf35939fe610","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"2dc26ede-d285-4cb8-a141-cf35939fe610","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"cf258840-f9d5-4269-b188-a2c608b486d6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ba6f3bc8-3e5b-438c-9bcd-5c42c35297ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"bc080de2-c3f9-4979-81ef-70446f5808d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"08dd7d55-8d78-4d63-854b-8068bdda5a3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"08dd7d55-8d78-4d63-854b-8068bdda5a3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"08dd7d55-8d78-4d63-854b-8068bdda5a3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"08dd7d55-8d78-4d63-854b-8068bdda5a3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ba6f3bc8-3e5b-438c-9bcd-5c42c35297ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ba6f3bc8-3e5b-438c-9bcd-5c42c35297ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ba6f3bc8-3e5b-438c-9bcd-5c42c35297ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"91b913ca-a007-4d77-89a8-08713ef2c182","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"47909ad7-086e-40f0-885b-b95673587843","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:25Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-12T08:57:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:26Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"f8fdfb94-c824-4068-aaef-14e547cf457f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:26Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:26Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/imagestreams.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:26Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.updateStatus"} {"level":"info","ts":"2026-07-12T08:57:26Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.reconcileHardwareProfiles"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.updateStatus"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"5a8114da-8a2a-4a17-9dc1-d26b3cae153a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"reconcile","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"apply","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.initialize"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.deployObservabilityManifests"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.configureDependencies"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"8bcdf68d-db17-41a4-b343-d8bb40266d0d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"reconcile","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"apply","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.initialize"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.configureDependencies"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.setKustomizedParams"} {"level":"info","ts":"2026-07-12T08:57:27Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:28Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:28Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.reconcileHardwareProfiles"} {"level":"info","ts":"2026-07-12T08:57:28Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.updateStatus"} {"level":"info","ts":"2026-07-12T08:57:28Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:28Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"ac564947-7921-4eda-a5ed-86dedd954711","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T08:57:29Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:29Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:29Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/imagestreams.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:29Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.updateStatus"} {"level":"info","ts":"2026-07-12T08:57:29Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T08:57:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9"} {"level":"info","ts":"2026-07-12T08:57:30Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"9877f4b4-fb2a-4ff4-b61b-d14dbad703a9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-12T08:57:37Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T08:58:34Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T08:58:48Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-kserve-raw-token-authentication","reconcileID":"2e005690-e91c-4ad9-913b-093e6e721cce"} {"level":"info","ts":"2026-07-12T08:58:48Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-kserve-raw-token-authentication","reconcileID":"4fe2caff-b136-4596-8b1c-41b22a5aac37"} {"level":"info","ts":"2026-07-12T08:58:48Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-kserve-raw-token-authentication","reconcileID":"608426e8-db7b-4a8e-b1b9-012a27a97458"} {"level":"error","ts":"2026-07-12T08:59:11Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:00:03Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:00:37Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-non-admin-raw","reconcileID":"e22227f5-46f7-4f80-bc86-e20e4377470a"} {"level":"info","ts":"2026-07-12T09:00:37Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-non-admin-raw","reconcileID":"bb254e49-a357-4edb-aef5-b0cd67554815"} {"level":"info","ts":"2026-07-12T09:00:37Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-non-admin-raw","reconcileID":"0ce6d667-bc43-4ba1-9f0f-fe017b301803"} {"level":"error","ts":"2026-07-12T09:01:03Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:01:26Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:01:37Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:01:52Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"sklearn-modelcar","reconcileID":"fa06f7cd-4180-4fe1-b01e-37d915ff8e99"} {"level":"info","ts":"2026-07-12T09:01:52Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"sklearn-modelcar","reconcileID":"dd2d2a0c-3647-4362-bfc2-5d917f297b09"} {"level":"info","ts":"2026-07-12T09:01:53Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"sklearn-modelcar","reconcileID":"6f239dd9-54bc-4b3e-b158-7a52204fb932"} {"level":"info","ts":"2026-07-12T09:02:29Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:02:32Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:02:53Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"onnx-standard-rest","reconcileID":"b61f0731-f770-490d-a005-db624c5f6a2d"} {"level":"info","ts":"2026-07-12T09:02:53Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"onnx-standard-rest","reconcileID":"28e57552-fc4b-483a-86c7-efe049527a02"} {"level":"info","ts":"2026-07-12T09:02:53Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"onnx-standard-rest","reconcileID":"2c4dd6fb-db3d-4da4-80d2-b24e4f7c3f11"} {"level":"info","ts":"2026-07-12T09:03:15Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ovms-smoke","reconcileID":"65597d42-0e97-4c2e-a5c2-9874b0c81b45"} {"level":"info","ts":"2026-07-12T09:03:15Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ovms-smoke","reconcileID":"d83200ee-0e7e-4095-99e1-1335a47d0634"} {"level":"info","ts":"2026-07-12T09:03:15Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ovms-smoke","reconcileID":"04b4f954-6e93-49ad-a10c-56be7418bf7d"} {"level":"error","ts":"2026-07-12T09:03:28Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:03:50Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"tensorflow-standard","reconcileID":"dd0d828a-32e6-49bf-81f9-c31e72b8e1f6"} {"level":"info","ts":"2026-07-12T09:03:50Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"tensorflow-standard","reconcileID":"73f21cf9-af7a-410d-8cb0-94f84fb92758"} {"level":"info","ts":"2026-07-12T09:03:50Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"tensorflow-standard","reconcileID":"2bb6366d-6270-4579-8568-d06d9d61c108"} {"level":"error","ts":"2026-07-12T09:04:06Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:05:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:05:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:05:50Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-12T09:05:55Z","msg":"reconcile","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"b5d731c4-ce7e-4f95-b252-36cc6ee261d8"} {"level":"info","ts":"2026-07-12T09:05:55Z","msg":"apply","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"b5d731c4-ce7e-4f95-b252-36cc6ee261d8"} {"level":"info","ts":"2026-07-12T09:05:55Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"b5d731c4-ce7e-4f95-b252-36cc6ee261d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.initialize"} {"level":"info","ts":"2026-07-12T09:05:55Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"b5d731c4-ce7e-4f95-b252-36cc6ee261d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:05:55Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"b5d731c4-ce7e-4f95-b252-36cc6ee261d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.createDefaultGroup"} {"level":"info","ts":"2026-07-12T09:05:55Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"b5d731c4-ce7e-4f95-b252-36cc6ee261d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/auth.managePermissions"} {"level":"info","ts":"2026-07-12T09:05:55Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"b5d731c4-ce7e-4f95-b252-36cc6ee261d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:05:55Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"b5d731c4-ce7e-4f95-b252-36cc6ee261d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:05:55Z","msg":"Executing action","controller":"auth","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Auth","Auth":{"name":"auth"},"namespace":"","name":"auth","reconcileID":"b5d731c4-ce7e-4f95-b252-36cc6ee261d8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-12T09:06:38Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:06:55Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"opt-125m-standard-cpu","reconcileID":"2b7cd82b-6cde-45d8-9f75-f2fed2bca310"} {"level":"info","ts":"2026-07-12T09:06:55Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"opt-125m-standard-cpu","reconcileID":"13bbfa9f-e5f1-4b45-8a3b-9adf03a366bc"} {"level":"info","ts":"2026-07-12T09:06:56Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"opt-125m-standard-cpu","reconcileID":"aa5ffe4f-f64b-434a-962b-2d5cd41b4a14"} {"level":"error","ts":"2026-07-12T09:07:16Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:08:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:08:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a344b7da-c642-4360-8bfc-cf9611023aa2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:09:13Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"024a2edd-d149-4090-8f0b-d8146fa87836","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:09:13Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"024a2edd-d149-4090-8f0b-d8146fa87836","name":"redhat-ods-applications"} {"level":"error","ts":"2026-07-12T09:09:32Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:10:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:10:31Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:10:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:10:56Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-12T09:11:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"95aed939-a262-4054-a9a8-56169f9edd89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-12T09:11:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:11:21Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"f0910b8e-ac40-4a56-a800-e72b582a39da","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-12T09:11:28Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:12:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-12T09:12:33Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:12:33Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e51e230-9b24-4c3f-8eca-18a0df9acdc7","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-12T09:12:44Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-12T09:12:48Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:12:48Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7a976b4a-668b-402f-af22-0febd8d3b1a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-12T09:13:39Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:14:31Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:15:15Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:15:17Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:15:37Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:15:57Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:16:44Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:17:31Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-4a1dd36b","reconcileID":"61cfe282-3454-4215-ba51-ddaa2e39d5d8"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9675021b-17e6-4b18-b9ad-e6d4abdaed7b"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9675021b-17e6-4b18-b9ad-e6d4abdaed7b"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9675021b-17e6-4b18-b9ad-e6d4abdaed7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9675021b-17e6-4b18-b9ad-e6d4abdaed7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9675021b-17e6-4b18-b9ad-e6d4abdaed7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9675021b-17e6-4b18-b9ad-e6d4abdaed7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9675021b-17e6-4b18-b9ad-e6d4abdaed7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-4a1dd36b","reconcileID":"9e45c029-b3c8-4ae4-a7a0-fb1c5fc99b80"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-4a1dd36b","reconcileID":"9b888fec-9661-4cd2-a5f2-3d6cf5312bef"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9675021b-17e6-4b18-b9ad-e6d4abdaed7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9675021b-17e6-4b18-b9ad-e6d4abdaed7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7c332bee-a216-40e7-a9ab-45317d34e0a3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"eee36a2e-fef2-4651-9c54-5c9d3e5cfcd2","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"eee36a2e-fef2-4651-9c54-5c9d3e5cfcd2","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:33Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"446c4f3e-3fb2-4116-b792-edd1fca1ca8a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"327ca193-da3e-46ef-b3e0-dd9b0aa1c79e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:34Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"373bc28b-e453-4523-84bf-ab1237248a4a","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"373bc28b-e453-4523-84bf-ab1237248a4a","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f99d168-19ba-45cb-98f4-6ce5b89bc63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"308d5dce-24c0-4364-8878-1648926e4c14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92d0fd39-b788-4235-98ee-bfb251042c68"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92d0fd39-b788-4235-98ee-bfb251042c68"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92d0fd39-b788-4235-98ee-bfb251042c68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92d0fd39-b788-4235-98ee-bfb251042c68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92d0fd39-b788-4235-98ee-bfb251042c68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92d0fd39-b788-4235-98ee-bfb251042c68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92d0fd39-b788-4235-98ee-bfb251042c68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92d0fd39-b788-4235-98ee-bfb251042c68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92d0fd39-b788-4235-98ee-bfb251042c68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b663de1c-89e1-49e9-854d-61eeabd6dea9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"95f5e1d3-4b20-4254-8f33-8f2a4dcad0d2","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"95f5e1d3-4b20-4254-8f33-8f2a4dcad0d2","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0996238c-0e66-4faf-84e5-241695fd45dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"94143792-6e4f-4254-af83-1f60dd3c4bb9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9cb0a7d-666a-49be-89c7-2bb609a262a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecbae062-4938-48e0-a47f-4b1ee9398f4e"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecbae062-4938-48e0-a47f-4b1ee9398f4e"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecbae062-4938-48e0-a47f-4b1ee9398f4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecbae062-4938-48e0-a47f-4b1ee9398f4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecbae062-4938-48e0-a47f-4b1ee9398f4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecbae062-4938-48e0-a47f-4b1ee9398f4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"60612cd1-c05f-47e5-a78d-c7e3d3df08a2","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecbae062-4938-48e0-a47f-4b1ee9398f4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"60612cd1-c05f-47e5-a78d-c7e3d3df08a2","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecbae062-4938-48e0-a47f-4b1ee9398f4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecbae062-4938-48e0-a47f-4b1ee9398f4e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e2c48faa-dec5-4004-bc73-ddb13f456fb1"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e2c48faa-dec5-4004-bc73-ddb13f456fb1"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e2c48faa-dec5-4004-bc73-ddb13f456fb1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e2c48faa-dec5-4004-bc73-ddb13f456fb1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e2c48faa-dec5-4004-bc73-ddb13f456fb1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e2c48faa-dec5-4004-bc73-ddb13f456fb1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e2c48faa-dec5-4004-bc73-ddb13f456fb1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e2c48faa-dec5-4004-bc73-ddb13f456fb1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e2c48faa-dec5-4004-bc73-ddb13f456fb1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e5264385-d636-4958-be21-c082c39a0e80","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:37Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"80ae22a2-99aa-48db-b612-b0ed8fbefb1d","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"80ae22a2-99aa-48db-b612-b0ed8fbefb1d","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ac31d9e-4cb1-47ab-86f8-012046b5f7e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"903d15cd-fa0d-4d41-a388-aaad8d39c352","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-cf933f2a","reconcileID":"fbcb7f57-677a-4f5c-aa0d-196a4b5614b9"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b6fc1491-35c2-4783-98ba-e7f01579d140"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b6fc1491-35c2-4783-98ba-e7f01579d140"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b6fc1491-35c2-4783-98ba-e7f01579d140","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b6fc1491-35c2-4783-98ba-e7f01579d140","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b6fc1491-35c2-4783-98ba-e7f01579d140","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b6fc1491-35c2-4783-98ba-e7f01579d140","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b6fc1491-35c2-4783-98ba-e7f01579d140","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b6fc1491-35c2-4783-98ba-e7f01579d140","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-cf933f2a","reconcileID":"dbcb0307-3a0f-4e1a-a623-dcaa4657c26a"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-cf933f2a","reconcileID":"fd293f01-26c7-43ac-96cf-fc140bb3120c"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b6fc1491-35c2-4783-98ba-e7f01579d140","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5718ccf7-a2fa-4b9f-a722-a00969cccee1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:39Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"1fd965a4-91c4-4dbe-9254-a3db67c9de00","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"1fd965a4-91c4-4dbe-9254-a3db67c9de00","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"e1485d3c-c17e-472d-8e94-ece3ba392023","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:39Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"7cc58952-d152-446b-8e20-14091c26f6e3","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"7cc58952-d152-446b-8e20-14091c26f6e3","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f82c3d66-84a1-4dc1-bf58-da06f53d6a52"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f82c3d66-84a1-4dc1-bf58-da06f53d6a52"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f82c3d66-84a1-4dc1-bf58-da06f53d6a52","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f82c3d66-84a1-4dc1-bf58-da06f53d6a52","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f82c3d66-84a1-4dc1-bf58-da06f53d6a52","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f82c3d66-84a1-4dc1-bf58-da06f53d6a52","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f82c3d66-84a1-4dc1-bf58-da06f53d6a52","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f82c3d66-84a1-4dc1-bf58-da06f53d6a52","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:39Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f82c3d66-84a1-4dc1-bf58-da06f53d6a52","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"03a85df2-e544-4f81-a453-9012f1e4d8ca","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"478e8c75-cf8c-49e0-a69a-93c758ca6e81","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"478e8c75-cf8c-49e0-a69a-93c758ca6e81","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9ca8bae5-6349-4944-9e69-6ad5b611d64e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dc6f71cd-4d1c-4fe3-afe7-d2a89200f860","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0715d49d-faee-47c2-9cba-9d623191b1b6"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0715d49d-faee-47c2-9cba-9d623191b1b6"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0715d49d-faee-47c2-9cba-9d623191b1b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0715d49d-faee-47c2-9cba-9d623191b1b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0715d49d-faee-47c2-9cba-9d623191b1b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0715d49d-faee-47c2-9cba-9d623191b1b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0715d49d-faee-47c2-9cba-9d623191b1b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0715d49d-faee-47c2-9cba-9d623191b1b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0715d49d-faee-47c2-9cba-9d623191b1b6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ef6594af-b3d6-4401-9e86-d893df0fb63a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:41Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"02986b3e-c6e8-4eab-bf7d-e4390c025726","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"02986b3e-c6e8-4eab-bf7d-e4390c025726","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8c362aa7-aa2b-4413-a58b-7df82ec09cd8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7aa654e6-4f8c-47a0-aede-9f97754f4d60","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d92fdf57-4c6a-4828-9652-1b0650608900","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ce9c5190-ce6e-4d9c-902b-6dbb5fb79d21","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ce9c5190-ce6e-4d9c-902b-6dbb5fb79d21","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"4e5be196-6dd4-4923-8f92-61622d7a0980","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cdfad0cb-7ec6-451c-9211-fc6bae6e60bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:17:43Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ed1e9457-f339-4ca7-951d-5dd0708e5ebb","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:17:43Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ed1e9457-f339-4ca7-951d-5dd0708e5ebb","name":"redhat-ods-applications"} ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-84fcf4d86-2j5cr.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf7988tz9x2.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-clkg6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-controller-manager-dd9f6cf9-l5vmt.log ==================== {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa"}} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:48Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"7300ed06-7642-4d68-b0ff-24e5934157e4"} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T08:58:48Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"error","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Error updating status after ingress reconcile failure","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:381\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T08:58:49Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"7300ed06-7642-4d68-b0ff-24e5934157e4","error":"fails to reconcile ingress: Route.route.openshift.io \"http-onnx\" not found","errorVerbose":"Route.route.openshift.io \"http-onnx\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:58:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:58:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:58:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"DeploymentReconciler","msg":"Deployment Updated","Diff":" v1.DeploymentSpec{\n \t... // 1 ignored field\n \tSelector: &{MatchLabels: {\"app\": \"isvc.http-onnx-predictor\"}},\n \tTemplate: v1.PodTemplateSpec{\n \t\tObjectMeta: v1.ObjectMeta{\n \t\t\t... // 9 identical fields\n \t\t\tDeletionGracePeriodSeconds: nil,\n \t\t\tLabels: {\"app\": \"isvc.http-onnx-predictor\", \"component\": \"predictor\", \"networking.kserve.io/visibility\": \"exposed\", \"serving.kserve.io/inferenceservice\": \"http-onnx\"},\n \t\t\tAnnotations: map[string]string{\n \t\t\t\t... // 4 identical entries\n \t\t\t\t\"prometheus.io/path\": \"/metrics\",\n \t\t\t\t\"prometheus.io/port\": \"8888\",\n- \t\t\t\t\"security.opendatahub.io/enable-auth\": \"true\",\n+ \t\t\t\t\"security.opendatahub.io/enable-auth\": \"false\",\n \t\t\t\t\"service.beta.openshift.io/serving-cert-secret-name\": \"http-onnx-predictor-serving-cert\",\n \t\t\t\t\"serving.kserve.io/deploymentMode\": \"Standard\",\n \t\t\t},\n \t\t\tOwnerReferences: nil,\n \t\t\tFinalizers: nil,\n \t\t\tManagedFields: nil,\n \t\t},\n \t\tSpec: {Volumes: {{Name: \"proxy-tls\", VolumeSource: {Secret: &{SecretName: \"http-onnx-predictor-serving-cert\", DefaultMode: &420}}}, {Name: \"kube-rbac-proxy-sar-config\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"http-onnx-kube-rbac-proxy-sar-config\"}, DefaultMode: &420}}}}, Containers: {{Name: \"kserve-container\", Image: \"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:\"..., Args: {\"--model_name=http-onnx\", \"--port=8001\", \"--rest_port=8888\", \"--model_path=/mnt/models\", ...}, Ports: {{ContainerPort: 8888, Protocol: \"TCP\"}}, ...}, {Name: \"kube-rbac-proxy\", Image: \"registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e\"..., Args: {\"--secure-listen-address=:8443\", \"--proxy-endpoints-port=8643\", \"--upstream=http://localhost:8888\", \"--auth-header-fields-enabled=true\", ...}, Ports: {{Name: \"https\", ContainerPort: 8443, Protocol: \"TCP\"}, {Name: \"proxy\", ContainerPort: 8643, Protocol: \"TCP\"}}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &30, ...},\n \t},\n \tStrategy: {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"25%\"}, MaxSurge: &{Type: 1, StrVal: \"25%\"}}},\n \tMinReadySeconds: 0,\n \t... // 3 identical fields\n }\n"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"error","ts":"2026-07-12T08:59:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T08:59:51Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"1d5f2d32-b570-4b69-847c-d5a123166c84","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:51Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"DeploymentReconciler","msg":"Deployment Updated","Diff":" v1.DeploymentSpec{\n \t... // 1 ignored field\n \tSelector: &{MatchLabels: {\"app\": \"isvc.http-onnx-predictor\"}},\n \tTemplate: v1.PodTemplateSpec{\n \t\tObjectMeta: v1.ObjectMeta{\n \t\t\t... // 9 identical fields\n \t\t\tDeletionGracePeriodSeconds: nil,\n \t\t\tLabels: {\"app\": \"isvc.http-onnx-predictor\", \"component\": \"predictor\", \"networking.kserve.io/visibility\": \"exposed\", \"serving.kserve.io/inferenceservice\": \"http-onnx\"},\n \t\t\tAnnotations: map[string]string{\n \t\t\t\t... // 4 identical entries\n \t\t\t\t\"prometheus.io/path\": \"/metrics\",\n \t\t\t\t\"prometheus.io/port\": \"8888\",\n- \t\t\t\t\"security.opendatahub.io/enable-auth\": \"false\",\n+ \t\t\t\t\"security.opendatahub.io/enable-auth\": \"true\",\n \t\t\t\t\"service.beta.openshift.io/serving-cert-secret-name\": \"http-onnx-predictor-serving-cert\",\n \t\t\t\t\"serving.kserve.io/deploymentMode\": \"Standard\",\n \t\t\t},\n \t\t\tOwnerReferences: nil,\n \t\t\tFinalizers: nil,\n \t\t\tManagedFields: nil,\n \t\t},\n \t\tSpec: {Volumes: {{Name: \"proxy-tls\", VolumeSource: {Secret: &{SecretName: \"http-onnx-predictor-serving-cert\", DefaultMode: &420}}}, {Name: \"kube-rbac-proxy-sar-config\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"http-onnx-kube-rbac-proxy-sar-config\"}, DefaultMode: &420}}}}, Containers: {{Name: \"kserve-container\", Image: \"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:\"..., Args: {\"--model_name=http-onnx\", \"--port=8001\", \"--rest_port=8888\", \"--model_path=/mnt/models\", ...}, Ports: {{ContainerPort: 8888, Protocol: \"TCP\"}}, ...}, {Name: \"kube-rbac-proxy\", Image: \"registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e\"..., Args: {\"--secure-listen-address=:8443\", \"--proxy-endpoints-port=8643\", \"--upstream=http://localhost:8888\", \"--auth-header-fields-enabled=true\", ...}, Ports: {{Name: \"https\", ContainerPort: 8443, Protocol: \"TCP\"}, {Name: \"proxy\", ContainerPort: 8643, Protocol: \"TCP\"}}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &30, ...},\n \t},\n \tStrategy: {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"25%\"}, MaxSurge: &{Type: 1, StrVal: \"25%\"}}},\n \tMinReadySeconds: 0,\n \t... // 3 identical fields\n }\n"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:58Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T08:59:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2"}} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:00Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx-2","reconcileID":"0dd72fdb-dc3a-471d-b08f-41c128954b41"} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:00Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"error","ts":"2026-07-12T09:00:01Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx-2","reconcileID":"0dd72fdb-dc3a-471d-b08f-41c128954b41","error":"fails to reconcile ingress: Route.route.openshift.io \"http-onnx-2\" not found","errorVerbose":"Route.route.openshift.io \"http-onnx-2\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"error","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx-2","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-2\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Error updating status after ingress reconcile failure","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-2\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-2\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:381\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:00:01Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx-2","reconcileID":"863797fb-c87f-4f86-a5c2-33c1821bd1e7","error":"fails to reconcile ingress: Route.route.openshift.io \"http-onnx-2\" not found","errorVerbose":"Route.route.openshift.io \"http-onnx-2\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:01Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:23Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:23Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:00:23Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:23Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:23Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:23Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"http-onnx"} {"level":"info","ts":"2026-07-12T09:00:24Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:24Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:24Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}}}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-raw","namespace":"test-non-admin-raw"},"namespace":"test-non-admin-raw","name":"http-onnx-raw","reconcileID":"d5270146-5fe4-4f65-aa40-1069dd692ef7"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"error","ts":"2026-07-12T09:00:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx-raw","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-raw\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:00:37Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-raw","namespace":"test-non-admin-raw"},"namespace":"test-non-admin-raw","name":"http-onnx-raw","reconcileID":"3f0edc36-fdc4-440b-a7f2-ab63c6c59bed","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-raw\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-raw\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:38Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:00:49Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:01:00Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:00Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:01:00Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:00Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:00Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:00Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}}}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"sklearn-modelcar","namespace":"sklearn-modelcar"},"namespace":"sklearn-modelcar","name":"sklearn-modelcar","reconcileID":"8a65038d-0d95-423a-8a2a-c7c6e97f77b5"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"error","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"sklearn-modelcar","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"sklearn-modelcar\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:01:53Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"sklearn-modelcar","namespace":"sklearn-modelcar"},"namespace":"sklearn-modelcar","name":"sklearn-modelcar","reconcileID":"ddce278f-e2c6-4252-976e-634a65ec8835","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"sklearn-modelcar\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"sklearn-modelcar\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:01:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:56Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:56Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:56Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:56Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:56Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:01:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:01:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:58Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:01:58Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:01:58Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:58Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:01:58Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:01:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:40Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:02:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:02:41Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:41Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:41Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"onnx"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"onnx"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"onnx","namespace":"onnx-standard-rest"},"namespace":"onnx-standard-rest","name":"onnx","reconcileID":"99671710-5b39-4693-947a-7a2e544ae74d"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"onnx"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"onnx"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"error","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"onnx","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"onnx\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:02:53Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"onnx","namespace":"onnx-standard-rest"},"namespace":"onnx-standard-rest","name":"onnx","reconcileID":"99671710-5b39-4693-947a-7a2e544ae74d","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"onnx\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"onnx\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"onnx"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"CredentialBuilder","msg":"Setting secret envs for s3","S3Secret":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"openvino-models-bucket-sa-dockercfg-j2pkk"} {"level":"info","ts":"2026-07-12T09:02:53Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"onnx","namespace":"onnx-standard-rest"},"namespace":"onnx-standard-rest","name":"onnx","reconcileID":"60b6335a-1356-43d7-a236-a5bd4ab005ed"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"error","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"onnx","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"onnx\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:02:53Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"onnx","namespace":"onnx-standard-rest"},"namespace":"onnx-standard-rest","name":"onnx","reconcileID":"6b7cbd40-0db7-4c09-acec-2441af9193bf","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"onnx\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"onnx\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:53Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:54Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:54Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:02:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:04Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:03:05Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:05Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"onnx"} {"level":"info","ts":"2026-07-12T09:03:05Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:05Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:05Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:05Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"},"namespace":"tensorflow-standard","name":"tensorflow-standard-rest","reconcileID":"6c96e2be-9341-4eda-acba-f14d9f638ea6"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"error","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"tensorflow-standard-rest","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"tensorflow-standard-rest\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:03:50Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"},"namespace":"tensorflow-standard","name":"tensorflow-standard-rest","reconcileID":"6c96e2be-9341-4eda-acba-f14d9f638ea6","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"tensorflow-standard-rest\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"tensorflow-standard-rest\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"CredentialBuilder","msg":"Setting secret envs for s3","S3Secret":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"triton-models-bucket-sa-dockercfg-qqdr7"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:50Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:51Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:52Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:03:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:06:36Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:06:46Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:46Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:06:46Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:46Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:46Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:46Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:46Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-standard-cpu","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-standard-cpu","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-standard-cpu","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-standard-cpu","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"},"namespace":"opt-125m-standard-cpu","name":"opt-125m-standard-cpu","reconcileID":"46e0bd1a-44ab-450b-aa7b-4b1fca587999"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} {"level":"error","ts":"2026-07-12T09:06:56Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"},"namespace":"opt-125m-standard-cpu","name":"opt-125m-standard-cpu","reconcileID":"46e0bd1a-44ab-450b-aa7b-4b1fca587999","error":"fails to reconcile ingress: Route.route.openshift.io \"opt-125m-standard-cpu\" not found","errorVerbose":"Route.route.openshift.io \"opt-125m-standard-cpu\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"CredentialBuilder","msg":"Setting secret envs for s3","S3Secret":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"models-bucket-sa-dockercfg-gfkbh"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-standard-cpu","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:06:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} {"level":"error","ts":"2026-07-12T09:06:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"opt-125m-standard-cpu","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"opt-125m-standard-cpu\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:06:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Error updating status after ingress reconcile failure","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"opt-125m-standard-cpu\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"opt-125m-standard-cpu\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:381\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:06:56Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"},"namespace":"opt-125m-standard-cpu","name":"opt-125m-standard-cpu","reconcileID":"f5d84a57-3de7-40b6-ae87-c884288857ea","error":"fails to reconcile ingress: Route.route.openshift.io \"opt-125m-standard-cpu\" not found","errorVerbose":"Route.route.openshift.io \"opt-125m-standard-cpu\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:06:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:06:59Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:07:02Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:16:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:16:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:16:57Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-standard-cpu"} {"level":"info","ts":"2026-07-12T09:16:57Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-standard-cpu","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-standard-cpu"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:16:57Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:16:57Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:16:57Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:16:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-standard-cpu"} ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-kqcfg.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-training-operator-54786944d6-rm4lg.log ==================== 2026-07-12T09:01:26Z INFO cert-rotation no cert refresh needed 2026-07-12T09:01:26Z INFO cert-rotation Ensuring CA cert {"name": "kubeflow-validator.training-operator.kubeflow.org", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "kubeflow-validator.training-operator.kubeflow.org", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"} ==================== WORKLOAD POD LOG: redhat-ods-applications/kuberay-operator-6586cdc875-j9bqf.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-56w7f.log ==================== {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-decode-template","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-decode-template","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"04891e42-53af-418b-84d0-af66850e2d1f","name":"v3-5-0-ea-2-kserve-config-llm-decode-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"8370cef3-5b27-4cf9-9618-5817f8408f04","name":"v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-prefill-template","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-prefill-template","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"84afc6d4-f7ff-4314-a6af-442012dbd5c6","name":"v3-5-0-ea-2-kserve-config-llm-prefill-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"c783724f-e24b-4649-b5ad-d41c5467904d","name":"v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-router-route","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-router-route","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"375b1ced-b1b8-4408-b667-4b2f42810462","name":"v3-5-0-ea-2-kserve-config-llm-router-route","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-scheduler","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-scheduler","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"d12096ad-9f03-44be-a206-992a53be9787","name":"v3-5-0-ea-2-kserve-config-llm-scheduler","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"9ed6a9a1-de58-4773-9970-5edd76841bf7","name":"v3-5-0-ea-2-kserve-config-llm-template","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-amd-rocm","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-amd-rocm","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"f2a984e6-62bf-43ec-9931-e69888ff58f1","name":"v3-5-0-ea-2-kserve-config-llm-template-amd-rocm","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-ppc64le","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-ppc64le","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"22bdd418-217c-4269-be40-7007b4ac82a8","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-ppc64le","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-s390x","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-s390x","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"a2cbe689-6fa5-4c20-8d8c-72f68ca128de","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-s390x","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-x86","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-x86","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"aa27d84c-fb60-4167-9e21-a077a960eea7","name":"v3-5-0-ea-2-kserve-config-llm-template-ibm-spyre-x86","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-intel-gaudi","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-intel-gaudi","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"b41d28c6-c4f7-423f-a06a-91035fc3e8d6","name":"v3-5-0-ea-2-kserve-config-llm-template-intel-gaudi","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-template-nvidia-cuda","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-template-nvidia-cuda","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"60142227-12d5-406f-b814-567727201014","name":"v3-5-0-ea-2-kserve-config-llm-template-nvidia-cuda","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:49Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-tracing","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-tracing","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"4935c480-aa70-4b27-9e63-4268f68bfa9f","name":"v3-5-0-ea-2-kserve-config-llm-tracing","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:56:50Z","logger":"admission","msg":"Validating LLMInferenceServiceConfig v1alpha2","webhookGroup":"serving.kserve.io","webhookKind":"LLMInferenceServiceConfig","LLMInferenceServiceConfig":{"name":"v3-5-0-ea-2-kserve-config-llm-worker-data-parallel","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"v3-5-0-ea-2-kserve-config-llm-worker-data-parallel","resource":{"group":"serving.kserve.io","version":"v1alpha2","resource":"llminferenceserviceconfigs"},"user":"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager","requestID":"2604d6d4-7acf-485e-856f-79c236efebdb","name":"v3-5-0-ea-2-kserve-config-llm-worker-data-parallel","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:59:36Z","logger":"controller-runtime.cache","msg":"Warning: VariantAutoscaling is deprecated and will be removed in a future release. Migrate to the annotation-based path (add llm-d.ai/managed=true to your HPA or ScaledObject). See docs/developer-guide/migrating-from-va-crd.md for migration steps."} {"level":"info","ts":"2026-07-12T09:07:35Z","logger":"controller-runtime.cache","msg":"Warning: VariantAutoscaling is deprecated and will be removed in a future release. Migrate to the annotation-based path (add llm-d.ai/managed=true to your HPA or ScaledObject). See docs/developer-guide/migrating-from-va-crd.md for migration steps."} {"level":"info","ts":"2026-07-12T09:17:11Z","logger":"controller-runtime.cache","msg":"Warning: VariantAutoscaling is deprecated and will be removed in a future release. Migrate to the annotation-based path (add llm-d.ai/managed=true to your HPA or ScaledObject). See docs/developer-guide/migrating-from-va-crd.md for migration steps."} ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-7c8d5f7bb-xfmb9.log ==================== {"level":"info","timestamp":"2026-07-12T08:56:49.421Z","caller":"config/cluster_config.go:103","message":"Created MaaSModelRef informer","watchNamespace":"ALL","gvr":"maas.opendatahub.io/v1alpha1, Resource=maasmodelrefs"} {"level":"info","timestamp":"2026-07-12T08:56:49.421Z","caller":"config/cluster_config.go:110","message":"Created MaaSSubscription informer","watchNamespace":"models-as-a-service","gvr":"maas.opendatahub.io/v1alpha1, Resource=maassubscriptions"} {"level":"info","timestamp":"2026-07-12T08:56:49.422Z","caller":"cmd/main.go:66","message":"Loading database connection URL from secret..."} {"level":"info","timestamp":"2026-07-12T08:56:49.522Z","caller":"cmd/main.go:173","message":"Connecting to PostgreSQL database...","tenant":"models-as-a-service"} {"level":"info","timestamp":"2026-07-12T08:56:49.522Z","caller":"cmd/main.go:102","message":"Metrics server starting","address":":9090"} {"level":"info","timestamp":"2026-07-12T08:56:49.539Z","caller":"api_keys/db_driver.go:91","message":"Database schema applied","version":5} {"level":"info","timestamp":"2026-07-12T08:56:49.539Z","caller":"api_keys/db_driver.go:58","message":"Connected to PostgreSQL database (schema applied)","tenant":"models-as-a-service"} {"level":"info","timestamp":"2026-07-12T08:56:49.539Z","caller":"cmd/main.go:180","message":"Starting informers and waiting for cache sync..."} {"level":"info","timestamp":"2026-07-12T08:56:49.639Z","caller":"cmd/main.go:184","message":"Informer caches synced successfully"} {"level":"info","timestamp":"2026-07-12T08:56:49.647Z","caller":"cmd/main.go:202","message":"Resolved gateway internal host for access probes","host":"maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local"} {"level":"info","timestamp":"2026-07-12T08:56:49.658Z","caller":"cmd/main.go:135","message":"Server starting","address":":8443","secure":true} [GIN] 2026/07/12 - 08:56:55 | 200 | 59.805µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:00 | 200 | 31.187µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:05 | 200 | 34.861µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:10 | 200 | 34.724µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:15 | 200 | 50.409µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:20 | 200 | 37.357µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:25 | 200 | 48.605µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:28 | 200 | 33.845µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:30 | 200 | 33.56µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:35 | 200 | 40.95µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:38 | 200 | 35.249µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:40 | 200 | 38.843µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:45 | 200 | 38.448µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:48 | 200 | 38.37µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:50 | 200 | 40.842µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:55 | 200 | 37.835µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:57:58 | 200 | 42.137µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:00 | 200 | 44.276µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:05 | 200 | 39.425µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:08 | 200 | 47.093µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:10 | 200 | 45.844µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:15 | 200 | 39.367µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:18 | 200 | 38.149µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:20 | 200 | 40.784µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:25 | 200 | 36.484µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:28 | 200 | 43.497µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:30 | 200 | 38.656µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:35 | 200 | 42.637µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:38 | 200 | 45.264µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:40 | 200 | 36.766µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:45 | 200 | 40.376µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:48 | 200 | 38.76µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:50 | 200 | 37.044µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:55 | 200 | 42.622µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:58:58 | 200 | 41.906µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:00 | 200 | 52.9µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:05 | 200 | 38.478µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:08 | 200 | 37.508µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:10 | 200 | 40.859µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:15 | 200 | 43.159µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:18 | 200 | 40.469µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:20 | 200 | 40.187µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:25 | 200 | 35.04µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:28 | 200 | 32.966µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:30 | 200 | 34.318µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:35 | 200 | 34.095µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:38 | 200 | 33.801µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:40 | 200 | 32.637µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:45 | 200 | 34.176µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:48 | 200 | 31.505µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:50 | 200 | 35.567µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:55 | 200 | 33.099µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 08:59:58 | 200 | 34.799µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:00 | 200 | 33.486µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:05 | 200 | 47.682µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:08 | 200 | 33.959µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:10 | 200 | 31.815µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:15 | 200 | 36.671µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:18 | 200 | 33.992µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:20 | 200 | 33.967µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:25 | 200 | 30.518µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:28 | 200 | 34.444µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:30 | 200 | 31.576µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:35 | 200 | 30.963µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:38 | 200 | 31.849µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:40 | 200 | 32.435µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:45 | 200 | 49.67µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:48 | 200 | 35.053µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:50 | 200 | 39.148µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:55 | 200 | 32.926µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:00:58 | 200 | 31.56µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:00 | 200 | 32.734µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:05 | 200 | 33.835µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:08 | 200 | 35.451µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:10 | 200 | 31.822µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:15 | 200 | 37.997µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:18 | 200 | 39.857µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:20 | 200 | 37.543µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:25 | 200 | 39.534µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:28 | 200 | 38.007µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:30 | 200 | 39.374µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:35 | 200 | 47.406µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:38 | 200 | 36.771µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:40 | 200 | 40.133µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:45 | 200 | 38.479µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:48 | 200 | 37.011µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:50 | 200 | 36.988µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:55 | 200 | 39.85µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:01:58 | 200 | 41.61µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:00 | 200 | 37.853µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:05 | 200 | 41.014µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:08 | 200 | 46.974µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:10 | 200 | 44.394µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:15 | 200 | 64.389µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:18 | 200 | 38.556µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:20 | 200 | 38.419µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:25 | 200 | 46.31µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:28 | 200 | 36.389µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:30 | 200 | 37.746µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:35 | 200 | 34.676µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:38 | 200 | 34.466µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:40 | 200 | 42.577µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:45 | 200 | 40.677µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:48 | 200 | 61.963µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:50 | 200 | 37.087µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:55 | 200 | 38.838µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:02:58 | 200 | 39.915µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:00 | 200 | 38.045µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:05 | 200 | 37.87µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:08 | 200 | 36.524µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:10 | 200 | 34.273µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:15 | 200 | 49.333µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:18 | 200 | 34.261µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:20 | 200 | 62.138µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:25 | 200 | 38.616µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:28 | 200 | 42.079µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:30 | 200 | 38.573µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:35 | 200 | 40.757µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:38 | 200 | 44.218µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:40 | 200 | 36.604µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:45 | 200 | 35.73µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:48 | 200 | 31.627µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:50 | 200 | 36.319µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:55 | 200 | 55.95µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:03:58 | 200 | 44.044µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:00 | 200 | 59.666µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:05 | 200 | 44.881µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:08 | 200 | 38.011µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:10 | 200 | 41.019µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:15 | 200 | 38.96µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:18 | 200 | 32.709µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:20 | 200 | 31.897µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:25 | 200 | 33.953µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:28 | 200 | 42.413µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:30 | 200 | 41.648µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:35 | 200 | 36.636µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:38 | 200 | 41.422µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:40 | 200 | 60.42µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:45 | 200 | 55.584µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:48 | 200 | 30.975µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:50 | 200 | 33.385µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:55 | 200 | 40.183µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:04:58 | 200 | 32.497µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:00 | 200 | 38.753µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:05 | 200 | 52.251µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:08 | 200 | 42.941µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:10 | 200 | 36.82µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:15 | 200 | 40.203µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:18 | 200 | 45.441µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:20 | 200 | 39.337µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:25 | 200 | 44.107µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:28 | 200 | 48.976µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:30 | 200 | 39.381µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:35 | 200 | 41.58µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:38 | 200 | 39.613µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:40 | 200 | 42.523µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:45 | 200 | 39.728µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:48 | 200 | 38.334µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:50 | 200 | 45.179µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:55 | 200 | 41.712µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:05:58 | 200 | 42.284µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:00 | 200 | 45.839µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:05 | 200 | 39.651µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:08 | 200 | 40.126µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:10 | 200 | 38.987µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:15 | 200 | 43.697µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:18 | 200 | 39.771µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:20 | 200 | 42.499µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:25 | 200 | 45.023µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:28 | 200 | 38.253µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:30 | 200 | 40.259µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:35 | 200 | 37.037µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:38 | 200 | 40.342µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:40 | 200 | 37.264µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:45 | 200 | 43.264µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:48 | 200 | 39.156µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:50 | 200 | 38.541µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:55 | 200 | 39.709µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:06:58 | 200 | 38.719µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:00 | 200 | 38.002µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:05 | 200 | 230.906µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:08 | 200 | 41.171µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:10 | 200 | 32.019µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:15 | 200 | 33.318µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:18 | 200 | 33.197µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:20 | 200 | 34.692µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:25 | 200 | 40.193µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:28 | 200 | 34.657µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:30 | 200 | 34.042µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:35 | 200 | 35.961µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:38 | 200 | 33.749µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:40 | 200 | 33.406µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:45 | 200 | 33.234µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:48 | 200 | 32.072µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:50 | 200 | 30.573µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:55 | 200 | 31.414µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:07:58 | 200 | 44.256µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:00 | 200 | 31.547µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:05 | 200 | 33.491µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:08 | 200 | 36.213µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:10 | 200 | 36.768µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:15 | 200 | 37.485µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:18 | 200 | 33.625µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:20 | 200 | 33µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:25 | 200 | 36.457µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:28 | 200 | 35.175µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:30 | 200 | 33.714µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:35 | 200 | 33.068µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:38 | 200 | 34.562µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:40 | 200 | 33.608µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:45 | 200 | 33.8µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:48 | 200 | 30.82µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:50 | 200 | 33.49µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:55 | 200 | 30.931µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:08:58 | 200 | 34.788µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:00 | 200 | 36.771µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:05 | 200 | 32.623µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:08 | 200 | 36.408µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:10 | 200 | 34.173µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:15 | 200 | 33.098µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:18 | 200 | 32.816µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:20 | 200 | 35.442µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:25 | 200 | 31.486µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:28 | 200 | 34.767µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:30 | 200 | 31.567µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:35 | 200 | 33.445µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:38 | 200 | 31.694µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:40 | 200 | 32.435µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:45 | 200 | 33.079µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:48 | 200 | 32.382µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:50 | 200 | 32.259µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:55 | 200 | 31.444µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:09:58 | 200 | 40.391µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:00 | 200 | 31.454µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:05 | 200 | 31.045µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:08 | 200 | 37.23µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:10 | 200 | 32.085µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:15 | 200 | 33.038µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:18 | 200 | 31.296µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:20 | 200 | 31.977µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:25 | 200 | 33.021µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:28 | 200 | 36.279µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:30 | 200 | 48.351µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:35 | 200 | 31.983µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:38 | 200 | 35.973µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:40 | 200 | 33.914µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:45 | 200 | 34.123µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:48 | 200 | 30.601µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:50 | 200 | 39.745µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:55 | 200 | 34.318µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:10:58 | 200 | 35.199µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:00 | 200 | 36.915µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:05 | 200 | 30.898µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:08 | 200 | 39.708µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:10 | 200 | 30.123µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:15 | 200 | 36.269µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:18 | 200 | 36.679µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:20 | 200 | 31.898µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:25 | 200 | 32.603µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:28 | 200 | 31.461µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:30 | 200 | 33.61µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:35 | 200 | 33.542µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:38 | 200 | 33.175µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:40 | 200 | 37.415µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:45 | 200 | 35.172µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:48 | 200 | 32.878µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:50 | 200 | 33.684µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:55 | 200 | 31.526µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:11:58 | 200 | 34.87µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:00 | 200 | 34.042µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:05 | 200 | 34.733µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:08 | 200 | 31.569µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:10 | 200 | 35.609µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:15 | 200 | 34.65µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:18 | 200 | 30.805µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:20 | 200 | 37.24µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:25 | 200 | 33.636µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:28 | 200 | 37.729µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:30 | 200 | 31.802µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:35 | 200 | 37.192µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:38 | 200 | 32.021µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:40 | 200 | 35.503µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:45 | 200 | 36.059µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:48 | 200 | 31.491µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:50 | 200 | 34.38µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:55 | 200 | 35.014µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:12:58 | 200 | 32.433µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:00 | 200 | 32.869µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:05 | 200 | 35.153µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:08 | 200 | 41.058µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:10 | 200 | 37.468µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:15 | 200 | 37.02µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:18 | 200 | 50.375µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:20 | 200 | 38.229µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:25 | 200 | 38.887µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:28 | 200 | 45.412µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:30 | 200 | 43.588µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:35 | 200 | 42.918µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:38 | 200 | 41.207µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:40 | 200 | 40.523µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:45 | 200 | 40.714µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:48 | 200 | 39.276µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:50 | 200 | 43.573µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:55 | 200 | 41.56µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:13:58 | 200 | 38.157µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:00 | 200 | 37.536µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:05 | 200 | 41.147µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:08 | 200 | 47.607µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:10 | 200 | 40.459µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:15 | 200 | 37.723µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:18 | 200 | 40.694µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:20 | 200 | 42.013µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:25 | 200 | 39.626µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:28 | 200 | 41.5µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:30 | 200 | 44.455µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:35 | 200 | 40.075µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:38 | 200 | 47.497µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:40 | 200 | 47.686µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:45 | 200 | 44.253µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:48 | 200 | 36.815µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:50 | 200 | 38.656µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:55 | 200 | 38.104µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:14:58 | 200 | 39.545µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:00 | 200 | 38.735µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:05 | 200 | 36.612µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:08 | 200 | 52.975µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:10 | 200 | 33.008µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:15 | 200 | 41.872µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:18 | 200 | 50.929µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:20 | 200 | 33.721µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:25 | 200 | 35.92µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:28 | 200 | 31.69µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:30 | 200 | 36.647µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:35 | 200 | 31.748µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:38 | 200 | 39.623µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:40 | 200 | 33.71µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:45 | 200 | 33.528µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:48 | 200 | 32.225µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:50 | 200 | 35.643µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:55 | 200 | 32.236µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:15:58 | 200 | 33.333µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:00 | 200 | 30.722µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:05 | 200 | 29.6µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:08 | 200 | 35.472µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:10 | 200 | 34.998µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:15 | 200 | 33.286µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:18 | 200 | 32.316µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:20 | 200 | 36.709µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:25 | 200 | 46.041µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:28 | 200 | 38.586µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:30 | 200 | 35.825µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:35 | 200 | 33.943µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:38 | 200 | 43.325µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:40 | 200 | 50.489µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:45 | 200 | 31.962µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:48 | 200 | 32.706µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:50 | 200 | 34.686µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:55 | 200 | 32.225µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:16:58 | 200 | 32.264µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:00 | 200 | 33.301µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:05 | 200 | 33.51µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:08 | 200 | 37.562µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:10 | 200 | 33.269µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:15 | 200 | 33.46µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:18 | 200 | 32.838µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:20 | 200 | 31.976µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:25 | 200 | 32.384µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:28 | 200 | 34.489µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:30 | 200 | 35.842µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:35 | 200 | 34.469µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:38 | 200 | 32.755µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:40 | 200 | 35.031µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:45 | 200 | 34.262µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:48 | 200 | 45.339µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:50 | 200 | 32.981µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:55 | 200 | 31.7µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:17:58 | 200 | 48.592µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:18:00 | 200 | 33.854µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:18:05 | 200 | 37.183µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:18:08 | 200 | 36.375µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:18:10 | 200 | 32.048µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:18:15 | 200 | 34.69µs | 10.128.0.2 | GET "/health" ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29728290-7sdjg.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29728305-2lkmp.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29728320-dzlkf.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29728335-zqlrm.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-controller-75bbfd978-nqzg4.log ==================== {"level":"info","ts":"2026-07-12T08:53:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:53:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:53:45Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T08:53:45Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T08:53:45Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"4d2e7819-9158-45dc-8660-05950362ec55","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T08:53:45Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T08:53:45Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T08:53:45Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T08:53:46Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"4d2e7819-9158-45dc-8660-05950362ec55","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:53:46Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"4d2e7819-9158-45dc-8660-05950362ec55","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:53:46Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T08:53:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T08:54:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:54:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:54:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:54:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:55:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:55:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:55:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:55:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:56:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:56:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:56:40Z","msg":"set Config owner reference on maas-controller Deployment","controller":"deployment","controllerGroup":"apps","controllerKind":"Deployment","Deployment":{"name":"maas-controller","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"maas-controller","reconcileID":"aea86e22-ad75-4041-b187-f60f55704066","deployment":{"name":"maas-controller","namespace":"redhat-ods-applications"}} {"level":"info","ts":"2026-07-12T08:56:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:56:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:57:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:57:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:57:24Z","msg":"set Config owner reference on maas-controller Deployment","controller":"deployment","controllerGroup":"apps","controllerKind":"Deployment","Deployment":{"name":"maas-controller","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"maas-controller","reconcileID":"b507aebc-87d2-4904-9871-b802d207ff19","deployment":{"name":"maas-controller","namespace":"redhat-ods-applications"}} {"level":"info","ts":"2026-07-12T08:57:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:57:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:58:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:58:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:58:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:58:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:58:46Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T08:58:46Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T08:58:46Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"0933bba9-f822-4e37-8a2a-264839c859bd","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T08:58:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T08:58:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T08:58:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T08:58:46Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"0933bba9-f822-4e37-8a2a-264839c859bd","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:58:46Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"0933bba9-f822-4e37-8a2a-264839c859bd","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T08:58:46Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T08:58:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T08:59:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:59:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T08:59:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:00:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:00:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:00:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:01:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:01:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:01:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:01:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:02:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:02:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:02:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:02:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:03:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:03:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:03:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:03:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:03:46Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:03:46Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:03:46Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"3e86173d-4043-442b-ad31-2db443fe8558","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T09:03:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:03:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:03:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:03:47Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"3e86173d-4043-442b-ad31-2db443fe8558","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:03:47Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"3e86173d-4043-442b-ad31-2db443fe8558","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:03:47Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:03:47Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:04:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:04:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:04:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:04:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:05:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:05:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:05:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:05:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:06:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:06:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:06:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:06:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:07:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:07:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:07:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:07:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:08:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:08:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:08:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:08:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:08:47Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:08:47Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:08:47Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"ba36cb75-a6b8-4f33-b9ea-8ea5a4c39fe3","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T09:08:47Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:08:47Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:08:47Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:08:47Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"ba36cb75-a6b8-4f33-b9ea-8ea5a4c39fe3","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:08:47Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"ba36cb75-a6b8-4f33-b9ea-8ea5a4c39fe3","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:08:47Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:08:47Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:09:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:09:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:09:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:09:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:10:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:10:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:10:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:10:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:11:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:11:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:11:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:11:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:12:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:12:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:12:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:12:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:13:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:13:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:13:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:13:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:13:47Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:13:47Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:13:47Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"da00447c-9ead-4359-8657-4e87f3819ae3","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T09:13:48Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:13:48Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:13:48Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:13:48Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"da00447c-9ead-4359-8657-4e87f3819ae3","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:13:48Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"da00447c-9ead-4359-8657-4e87f3819ae3","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:13:48Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:13:48Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:14:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:14:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:14:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:14:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:15:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:15:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:15:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:15:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:16:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:16:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:16:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:16:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:17:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:17:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:17:32Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:32Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:32Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"586f56b4-81b2-4136-84a5-6193a0eecca9","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","tenantID":"e2e-aigw-4a1dd36b","subscriptionNamespace":"ai-tenant-e2e-aigw-4a1dd36b","gatewayName":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:32Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:32Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:32Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"586f56b4-81b2-4136-84a5-6193a0eecca9","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"586f56b4-81b2-4136-84a5-6193a0eecca9","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:34Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:34Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:34Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:34Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:34Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"6bc00d99-0f32-49b0-ae83-7afccd025a80","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","tenantID":"e2e-aigw-4a1dd36b","subscriptionNamespace":"ai-tenant-e2e-aigw-4a1dd36b","gatewayName":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:34Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:34Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:34Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"6bc00d99-0f32-49b0-ae83-7afccd025a80","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"6bc00d99-0f32-49b0-ae83-7afccd025a80","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:35Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:35Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"error","ts":"2026-07-12T09:17:35Z","msg":"Reconciler error","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"6bc00d99-0f32-49b0-ae83-7afccd025a80","error":"Operation cannot be fulfilled on tenants.maas.opendatahub.io \"default-tenant\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"} {"level":"info","ts":"2026-07-12T09:17:35Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:35Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:35Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"567fcc71-328f-4fc1-bce1-5313752cfa1c","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","tenantID":"e2e-aigw-4a1dd36b","subscriptionNamespace":"ai-tenant-e2e-aigw-4a1dd36b","gatewayName":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:35Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:35Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:35Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"567fcc71-328f-4fc1-bce1-5313752cfa1c","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"567fcc71-328f-4fc1-bce1-5313752cfa1c","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"15c62644-cd47-42f2-9c90-fa45795c4eee","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","tenantID":"e2e-aigw-4a1dd36b","subscriptionNamespace":"ai-tenant-e2e-aigw-4a1dd36b","gatewayName":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"15c62644-cd47-42f2-9c90-fa45795c4eee","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"15c62644-cd47-42f2-9c90-fa45795c4eee","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Cleaning up MaaSAuthPolicy CRs","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"4edb356a-10f8-43fb-a213-0894d2b2f095","namespace":"ai-tenant-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Cleaned up MaaSAuthPolicy CRs","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"4edb356a-10f8-43fb-a213-0894d2b2f095","count":0} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-4a1dd36b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-4a1dd36b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-4a1dd36b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"tenantIdentifier":"e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Cleaning up per-tenant maas-api resources","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"4edb356a-10f8-43fb-a213-0894d2b2f095","tenant":"e2e-aigw-4a1dd36b","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:36Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-4a1dd36b","deploymentName":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"4edb356a-10f8-43fb-a213-0894d2b2f095","gvk":"Deployment","name":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:36Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"4edb356a-10f8-43fb-a213-0894d2b2f095","gvk":"Service","name":"maas-api-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"4edb356a-10f8-43fb-a213-0894d2b2f095","gvk":"HTTPRoute","name":"maas-api-e2e-aigw-4a1dd36b-route"} {"level":"info","ts":"2026-07-12T09:17:37Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-4a1dd36b"},"namespace":"ai-tenant-e2e-aigw-4a1dd36b","name":"default-tenant","reconcileID":"4edb356a-10f8-43fb-a213-0894d2b2f095","gvk":"CronJob","name":"maas-api-key-cleanup-e2e-aigw-4a1dd36b"} {"level":"info","ts":"2026-07-12T09:17:38Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-cf933f2a/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-cf933f2a","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-cf933f2a","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-cf933f2a"},"tenantIdentifier":"e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:38Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-cf933f2a/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-cf933f2a","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-cf933f2a","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-cf933f2a"},"tenantIdentifier":"e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:38Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"58ca34c9-2fda-45b0-90e2-e657c968309d","tenant":"ai-tenant-e2e-aigw-cf933f2a/default-tenant","tenantID":"e2e-aigw-cf933f2a","subscriptionNamespace":"ai-tenant-e2e-aigw-cf933f2a","gatewayName":"e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:38Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-cf933f2a","deploymentName":"maas-api-e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:38Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-cf933f2a","deploymentName":"maas-api-e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:38Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-cf933f2a","deploymentName":"maas-api-e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"58ca34c9-2fda-45b0-90e2-e657c968309d","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"58ca34c9-2fda-45b0-90e2-e657c968309d","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:40Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-cf933f2a/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-cf933f2a","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-cf933f2a","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-cf933f2a"},"tenantIdentifier":"e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:40Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-cf933f2a","deploymentName":"maas-api-e2e-aigw-cf933f2a"} {"level":"error","ts":"2026-07-12T09:17:40Z","msg":"Reconciler error","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"58ca34c9-2fda-45b0-90e2-e657c968309d","error":"Operation cannot be fulfilled on tenants.maas.opendatahub.io \"default-tenant\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"} {"level":"info","ts":"2026-07-12T09:17:40Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-cf933f2a/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-cf933f2a","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-cf933f2a","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-cf933f2a"},"tenantIdentifier":"e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:40Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-cf933f2a/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-cf933f2a","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-cf933f2a","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-cf933f2a"},"tenantIdentifier":"e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Cleaning up MaaSAuthPolicy CRs","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"66564ade-3ebe-49f9-99f4-6ffeda1e0810","namespace":"ai-tenant-e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Cleaned up MaaSAuthPolicy CRs","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"66564ade-3ebe-49f9-99f4-6ffeda1e0810","count":0} {"level":"info","ts":"2026-07-12T09:17:40Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-cf933f2a/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-cf933f2a","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-cf933f2a","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-cf933f2a"},"tenantIdentifier":"e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:40Z","msg":"Cleaning up per-tenant maas-api resources","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"66564ade-3ebe-49f9-99f4-6ffeda1e0810","tenant":"e2e-aigw-cf933f2a","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:17:40Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-cf933f2a","deploymentName":"maas-api-e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"66564ade-3ebe-49f9-99f4-6ffeda1e0810","gvk":"Deployment","name":"maas-api-e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"66564ade-3ebe-49f9-99f4-6ffeda1e0810","gvk":"Service","name":"maas-api-e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"66564ade-3ebe-49f9-99f4-6ffeda1e0810","gvk":"HTTPRoute","name":"maas-api-e2e-aigw-cf933f2a-route"} {"level":"info","ts":"2026-07-12T09:17:41Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-cf933f2a"},"namespace":"ai-tenant-e2e-aigw-cf933f2a","name":"default-tenant","reconcileID":"66564ade-3ebe-49f9-99f4-6ffeda1e0810","gvk":"CronJob","name":"maas-api-key-cleanup-e2e-aigw-cf933f2a"} {"level":"info","ts":"2026-07-12T09:17:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:17:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:18:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:18:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-postgres-bb9d55498-x2zxw.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/mlflow-operator-controller-manager-76556d4b59-tmg29.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-ccb5b4bbc-w9kcp.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-serving-api-5679d6b4f-ww2g7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebook-controller-deployment-647b7d4d45-p44z4.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-model-controller-65fdb9c5c-rncwr.log ==================== 2026-07-12T08:56:21Z DEBUG Reconciling EnvoyFilter for Gateway {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "59805440-295d-4073-baa8-65b50c37749c", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:56:21Z INFO Creating EnvoyFilter {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "59805440-295d-4073-baa8-65b50c37749c", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress", "name": "maas-default-gateway-authn-ssl"} 2026-07-12T08:56:21Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "59805440-295d-4073-baa8-65b50c37749c", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:56:21Z DEBUG Reconciling EnvoyFilter for Gateway {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "ff9cde03-c69a-4894-90e4-267364363b6d", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:56:21Z INFO Updating EnvoyFilter {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "ff9cde03-c69a-4894-90e4-267364363b6d", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress", "name": "maas-default-gateway-authn-ssl"} 2026-07-12T08:56:21Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "ff9cde03-c69a-4894-90e4-267364363b6d", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:56:21Z DEBUG Reconciling EnvoyFilter for Gateway {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "93a3b3a7-58a4-4ed2-aa76-1b8879a8914a", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:56:21Z INFO Updating EnvoyFilter {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "93a3b3a7-58a4-4ed2-aa76-1b8879a8914a", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress", "name": "maas-default-gateway-authn-ssl"} 2026-07-12T08:56:21Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "93a3b3a7-58a4-4ed2-aa76-1b8879a8914a", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:56:21Z DEBUG Reconciling EnvoyFilter for Gateway {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "ac96e977-79ee-4838-af55-e0341f56db2d", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:56:21Z INFO Updating EnvoyFilter {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "ac96e977-79ee-4838-af55-e0341f56db2d", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress", "name": "maas-default-gateway-authn-ssl"} 2026-07-12T08:56:22Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "ac96e977-79ee-4838-af55-e0341f56db2d", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:56:45Z DEBUG Reconciling EnvoyFilter for Gateway {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "efd868a8-62de-4f5e-9e6d-3d78a4c30741", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:56:45Z INFO Updating EnvoyFilter {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "efd868a8-62de-4f5e-9e6d-3d78a4c30741", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress", "name": "maas-default-gateway-authn-ssl"} 2026-07-12T08:56:45Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "efd868a8-62de-4f5e-9e6d-3d78a4c30741", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T08:58:48Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "c1ad709e-f3b4-4e95-808b-a33f5a13147f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T08:58:48Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "c1ad709e-f3b4-4e95-808b-a33f5a13147f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "c3fe1904-b2e4-4442-b9c5-a844fb46cb45", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4c7d4a43-d840-4789-aa5c-6b572c9e835a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "7936b7c1-e3e4-4a36-95c8-c8d305d58223", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T08:58:48Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "7936b7c1-e3e4-4a36-95c8-c8d305d58223", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "d0faa9dc-14b3-4288-9190-84cd5716a192", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T08:58:48Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "d0faa9dc-14b3-4288-9190-84cd5716a192", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1875fd75-9a23-4888-93e7-6916d69a1b74", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "6a615df8-c268-43dc-9db3-793edb930cbd", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z INFO Multi Node reconciling. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z INFO Reconcile event triggered by serving runtime: http-mnist-runtime 2026-07-12T08:58:48Z INFO No InferenceServices found for Serving Runtime: http-mnist-runtime 2026-07-12T08:58:48Z INFO Creating Storage Config Secret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "ci-bucket-secret", "reconcileID": "3b891876-70c0-438e-8c9a-3dfaeacef791", "Secret": "ci-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z INFO Reconcile event triggered by Secret: ci-bucket-secret 2026-07-12T08:58:48Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T08:58:48Z INFO Reconciling Storage Config Secret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"models-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "models-bucket-secret", "reconcileID": "1fb2a93b-3e45-49f2-9de6-1e872432c79f", "Secret": "models-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z INFO Reconcile event triggered by Secret: models-bucket-secret 2026-07-12T08:58:48Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T08:58:48Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T08:58:48Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "http-onnx", "name": "http-onnx"} 2026-07-12T08:58:48Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx"} 2026-07-12T08:58:48Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T08:58:48Z INFO inferenceservice-resource Namespace is not protected {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx"} 2026-07-12T08:58:48Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z INFO KubeAPIWarningLogger metadata.finalizers: "odh.inferenceservice.finalizers": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers 2026-07-12T08:58:48Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator"} 2026-07-12T08:58:48Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z ERROR Failed to fetch service for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "InferenceService": "http-onnx"} github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing /workspace/internal/controller/serving/inferenceservice_controller.go:164 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/inferenceservice_controller.go:173 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T08:58:48Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-metrics"} 2026-07-12T08:58:48Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-metrics"} 2026-07-12T08:58:48Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-metrics-dashboard"} 2026-07-12T08:58:48Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:48Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:48Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:48Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:48Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:48Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:48Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:48Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:48Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO Multi Node reconciled successfully. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "Namespace": "rhoai-model-registries"} 2026-07-12T08:58:49Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "94bb341a-5541-4c81-b0bd-63ef196fa7b3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx"} 2026-07-12T08:58:49Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:58:49Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:49Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "Namespace": "rhoai-model-registries"} 2026-07-12T08:58:49Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "df02902b-b0dd-4a7f-be42-2142262accc1", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:58:49Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:58:49Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:49Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "Namespace": "rhoai-model-registries"} 2026-07-12T08:58:49Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "0ef6ab33-c337-440e-97cf-b7a0d9a90881", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:58:49Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:58:49Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:49Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:49Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:49Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "Namespace": "rhoai-model-registries"} 2026-07-12T08:58:49Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "310b5410-ce38-4f1c-a2a0-b8be7e652555", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:58:57Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:58:57Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:57Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:57Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "Namespace": "rhoai-model-registries"} 2026-07-12T08:58:57Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "789f7e63-8ae6-4351-ab47-0e9d7814db11", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:58:58Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:58:58Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:58Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:58:58Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:58:58Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:58:58Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "Namespace": "rhoai-model-registries"} 2026-07-12T08:58:58Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a4dd425c-ff07-4193-ae90-9b40a9a40b31", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:43Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:43Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:43Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:43Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:43Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:43Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:43Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ca1df324-a1d7-49f7-aa2f-2b8cc6ea68f6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "delete": "test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator"} 2026-07-12T08:59:51Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:51Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:51Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:51Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:51Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4350c204-151b-4786-a074-3273c02fec86", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:51Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:51Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:51Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:51Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:51Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "b99842de-d21e-4f3a-9f45-2e502a0a663a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:51Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:52Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:52Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:52Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:52Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "3a0a0ea1-d5c0-4d41-8b4b-98ac37a5f95c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:52Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:52Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:52Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:52Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "8c3a4191-3a89-4841-a3a8-04ff6e921a09", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:52Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:52Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:52Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:52Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:52Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:52Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7d923611-252b-4fb6-a521-8f4896ffcdcf", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:53Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:53Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:53Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:53Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7af8eeb2-9c95-4f6b-9e25-21a67ce8ccae", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:54Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:54Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:54Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:54Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:54Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:54Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:54Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "af703ab9-b8e0-4595-8929-a7d334050d72", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator"} 2026-07-12T08:59:58Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:58Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:58Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:58Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:58Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:58Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:59Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "eb2dc5e9-36cd-46e3-b42f-3b73894074a7", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:59Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:59Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:59Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:59Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "6be22d43-ead6-4431-b04c-06e6869ba85f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T08:59:59Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T08:59:59Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T08:59:59Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T08:59:59Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T08:59:59Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "Namespace": "rhoai-model-registries"} 2026-07-12T08:59:59Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "7cf25d1f-fd38-4492-8d32-c9d25b73d1ad", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:00Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "http-onnx-2", "name": "http-onnx-2"} 2026-07-12T09:00:00Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx-2"} 2026-07-12T09:00:00Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:00:00Z INFO inferenceservice-resource Namespace is not protected {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx-2"} 2026-07-12T09:00:00Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:00Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:00Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:00Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:00Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "test-kserve-raw-token-authentication-models-bucket-sa-2-auth-delegator"} 2026-07-12T09:00:01Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z ERROR Failed to fetch service for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "InferenceService": "http-onnx-2"} github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing /workspace/internal/controller/serving/inferenceservice_controller.go:164 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/inferenceservice_controller.go:173 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:00:01Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-2-metrics"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-2-metrics"} 2026-07-12T09:00:01Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "fa2a76ac-89b7-426b-844d-6aedfeb59be9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-2"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:00:01Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "903c88b1-6879-4960-9ab8-2160318ede2c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:00:01Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "a9875a73-e99e-42cf-93ce-e1cb97b95bb4", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:00:01Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "6200914e-3e99-484d-ba6d-30628cd8c8f9", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:00:01Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:01Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:01Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "25378af3-3d04-4064-8ec4-a21ff485f20b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:00:02Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:00:02Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:02Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:02Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:02Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:02Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:02Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:02Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3668ec22-b26d-4f45-b90b-132292255ef7", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:00:04Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:00:04Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:04Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:04Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:04Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:04Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:04Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:04Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "64af3fc7-f780-4bfe-bc81-56310d5d6849", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:00:13Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:00:13Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:13Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:13Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:13Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:13Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:13Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:13Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:13Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:13Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:13Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:13Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "57548f12-6f80-45b8-b4b7-f434aeb48296", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2"} 2026-07-12T09:00:23Z DEBUG Deleting ClusterRoleBinding test-kserve-raw-token-authentication-models-bucket-sa-2-auth-delegator in namespace test-kserve-raw-token-authentication {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:23Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:23Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d541d6e5-ed8a-493f-b610-6e923090b707", "error": "InferenceService.serving.kserve.io \"http-onnx-2\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:00:23Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2"} 2026-07-12T09:00:23Z DEBUG Deleting ClusterRoleBinding test-kserve-raw-token-authentication-models-bucket-sa-2-auth-delegator in namespace test-kserve-raw-token-authentication {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:23Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:23Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:23Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "d3235a02-74a9-48da-be9b-b68751959815", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3bb06ad1-923f-4394-bf51-94e9e08c71b2", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3bb06ad1-923f-4394-bf51-94e9e08c71b2", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:23Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3bb06ad1-923f-4394-bf51-94e9e08c71b2", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "19bdeb76-4f95-487e-9e02-25f0a817e807", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:23Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "19bdeb76-4f95-487e-9e02-25f0a817e807", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:23Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "19bdeb76-4f95-487e-9e02-25f0a817e807", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx"} 2026-07-12T09:00:24Z DEBUG Deleting ClusterRoleBinding test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator in namespace test-kserve-raw-token-authentication {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:24Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "d2404f33-20de-4af1-b327-e84789a389bc", "error": "InferenceService.serving.kserve.io \"http-onnx\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:00:24Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx"} 2026-07-12T09:00:24Z DEBUG Deleting ClusterRoleBinding test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator in namespace test-kserve-raw-token-authentication {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:00:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:00:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:24Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "ab343782-d14b-4681-9f6a-e618f13a904c", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21400fbb-6f91-4e16-bf52-7f0c3e4fbad9", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21400fbb-6f91-4e16-bf52-7f0c3e4fbad9", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21400fbb-6f91-4e16-bf52-7f0c3e4fbad9", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21400fbb-6f91-4e16-bf52-7f0c3e4fbad9", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:24Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21400fbb-6f91-4e16-bf52-7f0c3e4fbad9", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "fb2f163a-5646-4863-9eb8-cbaa98459ae8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "fb2f163a-5646-4863-9eb8-cbaa98459ae8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "fb2f163a-5646-4863-9eb8-cbaa98459ae8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:24Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "fb2f163a-5646-4863-9eb8-cbaa98459ae8", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:24Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "fb2f163a-5646-4863-9eb8-cbaa98459ae8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:25Z INFO Reconcile event triggered by Secret: models-bucket-secret 2026-07-12T09:00:25Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"models-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "models-bucket-secret", "reconcileID": "918589c7-dd2b-475d-93ef-b8678128d623", "Secret": "models-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:25Z INFO Reconciling Storage Config Secret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"models-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "models-bucket-secret", "reconcileID": "918589c7-dd2b-475d-93ef-b8678128d623", "Secret": "models-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:25Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:00:25Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:00:25Z INFO Reconcile event triggered by Secret: ci-bucket-secret 2026-07-12T09:00:25Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "ci-bucket-secret", "reconcileID": "8b011320-a826-4418-a2d2-18338e386e30", "Secret": "ci-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:25Z INFO No data connections found in namespace {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "ci-bucket-secret", "reconcileID": "8b011320-a826-4418-a2d2-18338e386e30", "Secret": "ci-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:25Z INFO Deleting StorageSecret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "ci-bucket-secret", "reconcileID": "8b011320-a826-4418-a2d2-18338e386e30", "Secret": "ci-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:25Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:00:25Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"storage-config","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "storage-config", "reconcileID": "a3f4780a-9ae5-4df1-9038-5c3a5e6142df", "Secret": "storage-config", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:25Z INFO No data connections found in namespace {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"storage-config","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "storage-config", "reconcileID": "a3f4780a-9ae5-4df1-9038-5c3a5e6142df", "Secret": "storage-config", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:25Z INFO Reconcile event triggered by serving runtime: http-mnist-runtime 2026-07-12T09:00:25Z INFO No InferenceServices found for Serving Runtime: http-mnist-runtime 2026-07-12T09:00:25Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:25Z INFO Multi Node reconciling. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:30Z INFO Multi Node reconciled successfully. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:33Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "99244466-a59a-493d-80bf-27c3a1e088e7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:33Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "99244466-a59a-493d-80bf-27c3a1e088e7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:00:33Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "99244466-a59a-493d-80bf-27c3a1e088e7", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-kserve-raw-token-authentication because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:00:33Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9ea0adcf-2cc2-4de4-9dde-491d2c9ad2b1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:33Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9ea0adcf-2cc2-4de4-9dde-491d2c9ad2b1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:00:33Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9ea0adcf-2cc2-4de4-9dde-491d2c9ad2b1", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-kserve-raw-token-authentication because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:00:33Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "a0545988-e8ff-4682-8b30-0a63d524bf6b", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:33Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "a0545988-e8ff-4682-8b30-0a63d524bf6b", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:00:33Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "a0545988-e8ff-4682-8b30-0a63d524bf6b", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-kserve-raw-token-authentication because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:00:33Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "37284a60-c7fa-4f45-befa-b13500742b55", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:33Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "37284a60-c7fa-4f45-befa-b13500742b55", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:33Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "599ba287-2a3d-456a-97b0-7d7a91bfd6df", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:33Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "599ba287-2a3d-456a-97b0-7d7a91bfd6df", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:33Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "da4c9be6-3509-41b1-9496-1788314fcac5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:33Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "da4c9be6-3509-41b1-9496-1788314fcac5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:00:37Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "04a2833e-9e44-4d5c-98b0-3ce3ed978b9f", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "eabddc47-aeb8-4f4f-b86c-affb26811f3e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:00:37Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "eabddc47-aeb8-4f4f-b86c-affb26811f3e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "9c795199-0d00-479d-b12e-bd290997d830", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:00:37Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "9c795199-0d00-479d-b12e-bd290997d830", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "8ad130d3-cce7-4b40-a13f-5237bcd90af0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "9fdb4e67-84c0-47f4-a9ea-d3ecaa70106f", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "d7d9e638-6363-405b-a0fe-d3ea4cd0fc1a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:00:37Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "d7d9e638-6363-405b-a0fe-d3ea4cd0fc1a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a2c66e4f-9ae0-4693-98d7-25725186d5ed", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z INFO Reconcile event triggered by serving runtime: http-mnist-runtime 2026-07-12T09:00:37Z INFO No InferenceServices found for Serving Runtime: http-mnist-runtime 2026-07-12T09:00:37Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z INFO Multi Node reconciling. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z INFO Creating Storage Config Secret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-unprivileged","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "ci-bucket-unprivileged", "reconcileID": "eb91f804-136a-4a4e-9c0e-31ffa008f6f0", "Secret": "ci-bucket-unprivileged", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z INFO Reconcile event triggered by Secret: ci-bucket-unprivileged 2026-07-12T09:00:37Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:00:37Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "http-onnx-raw", "name": "http-onnx-raw"} 2026-07-12T09:00:37Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "test-non-admin-raw", "isvc": "http-onnx-raw"} 2026-07-12T09:00:37Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "test-non-admin-raw", "isvc": "http-onnx-raw", "namespace": "test-non-admin-raw", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:00:37Z INFO inferenceservice-resource Namespace is not protected {"namespace": "test-non-admin-raw", "isvc": "http-onnx-raw"} 2026-07-12T09:00:37Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "create": "http-onnx-raw-metrics"} 2026-07-12T09:00:37Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "create": "http-onnx-raw-metrics"} 2026-07-12T09:00:37Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "create": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:00:37Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:37Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:37Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:37Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:37Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca3295ae-c159-4215-806b-c02fe5a1b32b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:37Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:00:38Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:38Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:38Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "2089b12c-81ab-4b1e-a720-0a6251b9ee31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:00:38Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:38Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:38Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f69cb756-6343-416b-9da7-5d743b769075", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:00:38Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:38Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:38Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "38f561e9-9263-4384-8121-43cc5932efc0", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:00:38Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:38Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:38Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:38Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:38Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "fc67a481-a334-43d4-882d-9b43d3c3a89b", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z INFO Multi Node reconciled successfully. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:00:40Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:40Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:40Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:40Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:40Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:40Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "bc2073a3-037b-4acd-a364-aabfb11d0e87", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:00:49Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:49Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:49Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:00:49Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:00:49Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:00:49Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "Namespace": "rhoai-model-registries"} 2026-07-12T09:00:49Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "10e19e2e-30f9-4fec-ae58-73de8c87e4b5", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "name": "http-onnx-raw"} 2026-07-12T09:01:00Z DEBUG Deleting ClusterRoleBinding test-non-admin-raw-default-auth-delegator in namespace test-non-admin-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:01:00Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "Namespace": "rhoai-model-registries"} 2026-07-12T09:01:00Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "7afc99c3-6d4a-4561-ba7f-e1d30259ce31", "error": "InferenceService.serving.kserve.io \"http-onnx-raw\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:01:00Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "name": "http-onnx-raw"} 2026-07-12T09:01:00Z DEBUG Deleting ClusterRoleBinding test-non-admin-raw-default-auth-delegator in namespace test-non-admin-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:01:00Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:01:00Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "Namespace": "rhoai-model-registries"} 2026-07-12T09:01:00Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "5310d56a-53a8-4ab7-a560-728b03cc02d8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "e5ba7d33-46b6-44ca-8bae-e9550bc15195", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "e5ba7d33-46b6-44ca-8bae-e9550bc15195", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "e5ba7d33-46b6-44ca-8bae-e9550bc15195", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "e5ba7d33-46b6-44ca-8bae-e9550bc15195", "Namespace": "rhoai-model-registries"} 2026-07-12T09:01:00Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "e5ba7d33-46b6-44ca-8bae-e9550bc15195", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca51bcf3-d5cd-47d2-adaf-4f18416c32b1", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca51bcf3-d5cd-47d2-adaf-4f18416c32b1", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca51bcf3-d5cd-47d2-adaf-4f18416c32b1", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:00Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca51bcf3-d5cd-47d2-adaf-4f18416c32b1", "Namespace": "rhoai-model-registries"} 2026-07-12T09:01:00Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "ca51bcf3-d5cd-47d2-adaf-4f18416c32b1", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:01Z INFO Reconcile event triggered by Secret: ci-bucket-unprivileged 2026-07-12T09:01:01Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-unprivileged","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "ci-bucket-unprivileged", "reconcileID": "9eee66a8-2dc9-4294-83d9-93691f00a385", "Secret": "ci-bucket-unprivileged", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:01Z INFO No data connections found in namespace {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-unprivileged","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "ci-bucket-unprivileged", "reconcileID": "9eee66a8-2dc9-4294-83d9-93691f00a385", "Secret": "ci-bucket-unprivileged", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:01Z INFO Deleting StorageSecret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-unprivileged","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "ci-bucket-unprivileged", "reconcileID": "9eee66a8-2dc9-4294-83d9-93691f00a385", "Secret": "ci-bucket-unprivileged", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:01Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:01:01Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"storage-config","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "storage-config", "reconcileID": "31156f4e-b57c-408a-a946-08d090afc88e", "Secret": "storage-config", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:01Z INFO No data connections found in namespace {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"storage-config","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "storage-config", "reconcileID": "31156f4e-b57c-408a-a946-08d090afc88e", "Secret": "storage-config", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:01Z INFO Reconcile event triggered by serving runtime: http-mnist-runtime 2026-07-12T09:01:01Z INFO No InferenceServices found for Serving Runtime: http-mnist-runtime 2026-07-12T09:01:01Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:01:01Z INFO Multi Node reconciling. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:01:04Z INFO Multi Node reconciled successfully. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:01:06Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a77f9e4c-e842-4aed-b141-584c11a31250", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:06Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a77f9e4c-e842-4aed-b141-584c11a31250", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:01:06Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a77f9e4c-e842-4aed-b141-584c11a31250", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-non-admin-raw because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:01:06Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5742a099-5274-4d7d-bd80-9180490695fa", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:06Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5742a099-5274-4d7d-bd80-9180490695fa", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:01:06Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5742a099-5274-4d7d-bd80-9180490695fa", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-non-admin-raw because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:01:06Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "77a7113a-c361-4ca2-9cc5-448ce4273f05", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:06Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "77a7113a-c361-4ca2-9cc5-448ce4273f05", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:01:06Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "77a7113a-c361-4ca2-9cc5-448ce4273f05", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-non-admin-raw because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:01:06Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "9a28032b-57e4-4aad-b7e9-54f5b89e1125", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:06Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "9a28032b-57e4-4aad-b7e9-54f5b89e1125", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:06Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "e0a1d917-509a-4971-bad6-d86486b95164", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:06Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "e0a1d917-509a-4971-bad6-d86486b95164", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:06Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "20af7b90-7731-4413-a5bb-e080932ec7e5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:06Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "20af7b90-7731-4413-a5bb-e080932ec7e5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:01:52Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "9d8a39b3-d555-4154-a1ab-a1f4e9b92459", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:52Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "480aab96-72e8-4948-9ec5-f4e72e44936f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:01:52Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "480aab96-72e8-4948-9ec5-f4e72e44936f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:52Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "e42f6346-2ab4-4960-abe3-d8a88a5cb145", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:01:53Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "e42f6346-2ab4-4960-abe3-d8a88a5cb145", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "90eab821-b0dd-4b81-8b12-1dca86358ca8", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:01:53Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "90eab821-b0dd-4b81-8b12-1dca86358ca8", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "f849b886-64dc-441a-8b20-c27ec9683f3e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:01:53Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "f849b886-64dc-441a-8b20-c27ec9683f3e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6a4f905b-9f3a-4129-9db0-0abb499f4eb9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO Reconcile event triggered by serving runtime: mlserver-runtime 2026-07-12T09:01:53Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO No InferenceServices found for Serving Runtime: mlserver-runtime 2026-07-12T09:01:53Z INFO Multi Node reconciling. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "sklearn-modelcar", "name": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "sklearn-modelcar", "isvc": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "sklearn-modelcar", "isvc": "sklearn-modelcar", "namespace": "sklearn-modelcar", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:01:53Z INFO inferenceservice-resource Namespace is not protected {"namespace": "sklearn-modelcar", "isvc": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "create": "sklearn-modelcar-metrics"} 2026-07-12T09:01:53Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "create": "sklearn-modelcar-metrics"} 2026-07-12T09:01:53Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "create": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:01:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "Namespace": "rhoai-model-registries"} 2026-07-12T09:01:53Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "df0bd1e7-83e1-40e2-99af-11e223198c0c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "ConfigMap:": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:01:53Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:01:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "Namespace": "rhoai-model-registries"} 2026-07-12T09:01:53Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "796432c5-362e-4d3f-9991-e550b1f7087f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "ConfigMap:": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:01:53Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:01:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "Namespace": "rhoai-model-registries"} 2026-07-12T09:01:53Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "7fd6dcbb-7380-4241-9a94-41980ba3287e", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "ConfigMap:": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:01:53Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:01:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:01:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "Namespace": "rhoai-model-registries"} 2026-07-12T09:01:53Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "eed4fb51-8dea-4d2c-993d-292ce11fa401", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:01:58Z INFO Multi Node reconciled successfully. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "ConfigMap:": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:02:40Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:02:40Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:40Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:40Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:02:40Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:40Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:40Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "96f7b2d1-5c2e-4218-b08d-24c1f696b20b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "name": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG Deleting ClusterRoleBinding sklearn-modelcar-default-auth-delegator in namespace sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:02:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:41Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "007291b8-a59b-4530-805d-4ae582bc7043", "error": "InferenceService.serving.kserve.io \"sklearn-modelcar\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:02:41Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "name": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG Deleting ClusterRoleBinding sklearn-modelcar-default-auth-delegator in namespace sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:02:41Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:02:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:41Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "93344f8b-e3ab-4055-b3c2-19fc4d0bc54d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "881dd013-ca8a-4706-8abc-46fceb75df3d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "881dd013-ca8a-4706-8abc-46fceb75df3d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "881dd013-ca8a-4706-8abc-46fceb75df3d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "881dd013-ca8a-4706-8abc-46fceb75df3d", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:41Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "881dd013-ca8a-4706-8abc-46fceb75df3d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "60e10b3c-4643-4de9-9965-7bd0214f6d81", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "60e10b3c-4643-4de9-9965-7bd0214f6d81", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "60e10b3c-4643-4de9-9965-7bd0214f6d81", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:41Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "60e10b3c-4643-4de9-9965-7bd0214f6d81", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:41Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "60e10b3c-4643-4de9-9965-7bd0214f6d81", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:42Z INFO Reconcile event triggered by serving runtime: mlserver-runtime 2026-07-12T09:02:42Z INFO No InferenceServices found for Serving Runtime: mlserver-runtime 2026-07-12T09:02:42Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:02:42Z INFO Multi Node reconciling. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:02:44Z INFO Multi Node reconciled successfully. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:02:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3ddd041c-aded-4727-99b7-a684ab24fe0d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:49Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3ddd041c-aded-4727-99b7-a684ab24fe0d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:02:49Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3ddd041c-aded-4727-99b7-a684ab24fe0d", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace sklearn-modelcar because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:02:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dd745b84-cdb7-430a-8ac6-b7c31af2afc7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:49Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dd745b84-cdb7-430a-8ac6-b7c31af2afc7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:02:49Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dd745b84-cdb7-430a-8ac6-b7c31af2afc7", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace sklearn-modelcar because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:02:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "7ef8998e-c390-4142-9cc4-9821b4a1f405", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:49Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "7ef8998e-c390-4142-9cc4-9821b4a1f405", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:02:49Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "7ef8998e-c390-4142-9cc4-9821b4a1f405", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace sklearn-modelcar because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:02:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "146f00b6-662f-406d-bd2c-0dd0d3528908", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:49Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "146f00b6-662f-406d-bd2c-0dd0d3528908", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "82c2e169-23a1-4233-9517-93ae3b1b6675", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:49Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "82c2e169-23a1-4233-9517-93ae3b1b6675", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2adca8ee-b60d-42f7-b350-0567c1319a1f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:49Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2adca8ee-b60d-42f7-b350-0567c1319a1f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:02:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "d39bacd1-7044-4b79-8af4-dc8b2615a717", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:02:53Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "d39bacd1-7044-4b79-8af4-dc8b2615a717", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "openshift-service-ca.crt", "reconcileID": "aec204f1-53cd-416a-84d0-235529f29a2c", "ConfigMap": "openshift-service-ca.crt", "namespace": "onnx-standard-rest", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:02:53Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "openshift-service-ca.crt", "reconcileID": "aec204f1-53cd-416a-84d0-235529f29a2c", "ConfigMap": "openshift-service-ca.crt", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5b263527-94b3-44bf-afef-c4f227f9ac86", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "openshift-service-ca.crt", "reconcileID": "ae844107-d10d-4350-80f0-6f27dee43a44", "ConfigMap": "openshift-service-ca.crt", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "8cfd78a8-6db7-4154-94e8-02da0c467738", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:02:53Z INFO Reconcile event triggered by serving runtime: openvino-runtime 2026-07-12T09:02:53Z INFO No InferenceServices found for Serving Runtime: openvino-runtime 2026-07-12T09:02:53Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO Multi Node reconciling. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "8cfd78a8-6db7-4154-94e8-02da0c467738", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1078d93a-e833-4a69-88b3-2ad285475e6e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "onnx", "name": "onnx"} 2026-07-12T09:02:53Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "onnx-standard-rest", "isvc": "onnx"} 2026-07-12T09:02:53Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "onnx-standard-rest", "isvc": "onnx", "namespace": "onnx-standard-rest", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:02:53Z INFO inferenceservice-resource Namespace is not protected {"namespace": "onnx-standard-rest", "isvc": "onnx"} 2026-07-12T09:02:53Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "create": "onnx-metrics"} 2026-07-12T09:02:53Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "create": "onnx-metrics"} 2026-07-12T09:02:53Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "create": "onnx-metrics-dashboard"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:53Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "46602ee6-58ca-4ae7-a580-1140276b04bc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:02:53Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:53Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8d41eb53-5d58-44af-9c61-1cc80da018de", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:02:53Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:53Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "b8804167-caac-47fd-a278-35c4153b442e", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:02:55Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:55Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:55Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:55Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:55Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8a6facdb-d773-488b-bb73-a062dae4b44b", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:02:56Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:56Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:02:56Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:02:56Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:56Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "Namespace": "rhoai-model-registries"} 2026-07-12T09:02:56Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "90d4e51f-7565-456f-aa17-1e017b8bdb69", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:02:57Z INFO Multi Node reconciled successfully. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:03:04Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:03:04Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:04Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:04Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:03:04Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:04Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:04Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "58eb3be3-07bc-4e91-89ab-a9bf4528022c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "name": "onnx"} 2026-07-12T09:03:05Z DEBUG Deleting ClusterRoleBinding onnx-standard-rest-openvino-models-bucket-sa-auth-delegator in namespace onnx-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:03:05Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:05Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d08cfe5d-8d67-46fe-86ca-a38b07327b1c", "error": "InferenceService.serving.kserve.io \"onnx\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:03:05Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "name": "onnx"} 2026-07-12T09:03:05Z DEBUG Deleting ClusterRoleBinding onnx-standard-rest-openvino-models-bucket-sa-auth-delegator in namespace onnx-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:03:05Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:03:05Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:05Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "5912fe86-dbd9-46b7-a461-2a7920c90fdc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "70da0223-7720-4427-957a-17113247adcc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "70da0223-7720-4427-957a-17113247adcc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "70da0223-7720-4427-957a-17113247adcc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "70da0223-7720-4427-957a-17113247adcc", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:05Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "70da0223-7720-4427-957a-17113247adcc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f1fae979-12cf-4434-a3cf-f35714d7f5a1", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f1fae979-12cf-4434-a3cf-f35714d7f5a1", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f1fae979-12cf-4434-a3cf-f35714d7f5a1", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:05Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f1fae979-12cf-4434-a3cf-f35714d7f5a1", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:05Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f1fae979-12cf-4434-a3cf-f35714d7f5a1", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:06Z INFO Reconcile event triggered by serving runtime: openvino-runtime 2026-07-12T09:03:06Z INFO No InferenceServices found for Serving Runtime: openvino-runtime 2026-07-12T09:03:06Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:03:06Z INFO Multi Node reconciling. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:03:09Z INFO Multi Node reconciled successfully. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:03:13Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6d975abd-36e2-4c1c-9fb7-26c50df4fc92", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:13Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6d975abd-36e2-4c1c-9fb7-26c50df4fc92", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:13Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6d975abd-36e2-4c1c-9fb7-26c50df4fc92", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace onnx-standard-rest because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:03:13Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4c658b70-44c3-4a68-ad10-4eefa3c68ad4", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:13Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4c658b70-44c3-4a68-ad10-4eefa3c68ad4", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:13Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4c658b70-44c3-4a68-ad10-4eefa3c68ad4", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace onnx-standard-rest because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:03:13Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "e551a28f-8cb2-4595-931c-2131e151a6a7", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:13Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "e551a28f-8cb2-4595-931c-2131e151a6a7", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:13Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "openshift-service-ca.crt", "reconcileID": "7e25ad93-28f2-42a0-b9b8-4279dd59b28b", "ConfigMap": "openshift-service-ca.crt", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:13Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "openshift-service-ca.crt", "reconcileID": "7e25ad93-28f2-42a0-b9b8-4279dd59b28b", "ConfigMap": "openshift-service-ca.crt", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:13Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "06b52756-629c-4279-aa8b-5eb5c9a81c82", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:13Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "06b52756-629c-4279-aa8b-5eb5c9a81c82", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:03:15Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "4a4ad901-c598-4fc6-a4fa-e1e9aa14d448", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:03:15Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "80ded047-1894-4486-b718-0f60633b572b", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:15Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "80ded047-1894-4486-b718-0f60633b572b", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:15Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "762f6381-6029-4c1b-a1ac-949f067be2c7", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:15Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "762f6381-6029-4c1b-a1ac-949f067be2c7", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:03:15Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ee50718c-9417-4571-acde-781cdeac8b9a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:15Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "aec0018e-c6b5-47cf-a728-cfac3937cfb0", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:03:15Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "462ea9d8-1cfd-481b-9b40-ebb5aa9364f7", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:15Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "462ea9d8-1cfd-481b-9b40-ebb5aa9364f7", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:15Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "84e1de36-26d5-4882-9220-0e6eeeb37560", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:46Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0f2a3276-752c-4571-a1f6-6bd3fc1da47d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:46Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0f2a3276-752c-4571-a1f6-6bd3fc1da47d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:46Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0f2a3276-752c-4571-a1f6-6bd3fc1da47d", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ovms-smoke because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:03:46Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fa2776ff-b810-4970-bb22-072bf51e44b8", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:46Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fa2776ff-b810-4970-bb22-072bf51e44b8", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:46Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fa2776ff-b810-4970-bb22-072bf51e44b8", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ovms-smoke because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:03:46Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "d37b7155-2567-4190-8862-162841267a2f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:46Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "d37b7155-2567-4190-8862-162841267a2f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:46Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "84aef9d0-ae9a-4c68-87ff-214209b0b211", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:03:46Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "84aef9d0-ae9a-4c68-87ff-214209b0b211", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:03:46Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "67d708e6-4c9a-4691-9720-cffb2e0d1818", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:46Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "67d708e6-4c9a-4691-9720-cffb2e0d1818", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "82ba33a7-20b9-46c7-a819-f1ff2f64170d", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "20c48dfe-9dfc-45f6-a5d3-f57eb01c66df", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:50Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "20c48dfe-9dfc-45f6-a5d3-f57eb01c66df", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "4a927f49-d524-408a-9065-18a136fff682", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:50Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "4a927f49-d524-408a-9065-18a136fff682", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f198346b-d1eb-4237-8770-86a3a990ef93", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "45bd64bb-1dcd-4c76-bb0c-a8261e9e2da5", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "0a14f9e3-3811-46fb-9f82-4eb8edabf90a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:03:50Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO Multi Node reconciling. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO Reconcile event triggered by serving runtime: triton-rest-runtime 2026-07-12T09:03:50Z INFO No InferenceServices found for Serving Runtime: triton-rest-runtime 2026-07-12T09:03:50Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "0a14f9e3-3811-46fb-9f82-4eb8edabf90a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4dff5516-a829-4e92-8ef0-b98f4555027e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "tensorflow-standard-rest", "name": "tensorflow-standard-rest"} 2026-07-12T09:03:50Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "tensorflow-standard", "isvc": "tensorflow-standard-rest"} 2026-07-12T09:03:50Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "tensorflow-standard", "isvc": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:03:50Z INFO inferenceservice-resource Namespace is not protected {"namespace": "tensorflow-standard", "isvc": "tensorflow-standard-rest"} 2026-07-12T09:03:50Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "create": "tensorflow-standard-rest-metrics"} 2026-07-12T09:03:50Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "create": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:50Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:50Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "43422a99-a619-4a92-bb44-7c3f0401364e", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:03:50Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:50Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:50Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "ce8f0ed5-1e5b-4bbf-abac-7211957ac574", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:03:50Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:50Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:50Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "83613cf6-49b0-43de-94b0-2ee2f3c7fe98", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:03:50Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:50Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:50Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:03:50Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:50Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "74508523-313a-4aba-800b-53608107194d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z INFO Multi Node reconciled successfully. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:03:52Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:52Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:52Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:52Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:52Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:03:52Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:52Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "5af3ba16-61d9-43f3-9bea-94e225c877b6", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:03:54Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:54Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:03:54Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:03:54Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:03:54Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "Namespace": "rhoai-model-registries"} 2026-07-12T09:03:54Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "0cd0911c-2348-4713-acff-f388c4c4fa17", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:06:36Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:06:36Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:36Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:36Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:36Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:36Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:36Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:06:36Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:36Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:36Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "cdb2dc93-49d4-484b-b433-f29d6a2b7fb0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest"} 2026-07-12T09:06:46Z DEBUG Deleting ClusterRoleBinding tensorflow-standard-triton-models-bucket-sa-auth-delegator in namespace tensorflow-standard {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:06:46Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:46Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "65cdd94e-ba04-4b8e-900e-41154f6c5d83", "error": "InferenceService.serving.kserve.io \"tensorflow-standard-rest\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:46Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest"} 2026-07-12T09:06:46Z DEBUG Deleting ClusterRoleBinding tensorflow-standard-triton-models-bucket-sa-auth-delegator in namespace tensorflow-standard {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:06:46Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:06:46Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:46Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "69c43660-d9cf-41ce-b261-2f58786ddc0d", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9f006508-7c91-4b2d-bca7-7833eab4aa52", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9f006508-7c91-4b2d-bca7-7833eab4aa52", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9f006508-7c91-4b2d-bca7-7833eab4aa52", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9f006508-7c91-4b2d-bca7-7833eab4aa52", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:46Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9f006508-7c91-4b2d-bca7-7833eab4aa52", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9ff6ac4c-0a29-4079-a62c-29b8be419486", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9ff6ac4c-0a29-4079-a62c-29b8be419486", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9ff6ac4c-0a29-4079-a62c-29b8be419486", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:06:46Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9ff6ac4c-0a29-4079-a62c-29b8be419486", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:46Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "9ff6ac4c-0a29-4079-a62c-29b8be419486", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:06:47Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:06:47Z INFO Reconcile event triggered by serving runtime: triton-rest-runtime 2026-07-12T09:06:47Z INFO Multi Node reconciling. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:06:47Z INFO No InferenceServices found for Serving Runtime: triton-rest-runtime 2026-07-12T09:06:49Z INFO Multi Node reconciled successfully. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d383a6c1-b871-4c4f-9a03-71a16f5ae08a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d383a6c1-b871-4c4f-9a03-71a16f5ae08a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:53Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d383a6c1-b871-4c4f-9a03-71a16f5ae08a", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "675c6a98-69bd-4848-8557-c8f301088cd2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "675c6a98-69bd-4848-8557-c8f301088cd2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:53Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "675c6a98-69bd-4848-8557-c8f301088cd2", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "437b3436-75cf-4fa2-b652-4286de765dca", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "437b3436-75cf-4fa2-b652-4286de765dca", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:53Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "437b3436-75cf-4fa2-b652-4286de765dca", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3e9f66a4-13c7-4391-964b-707e2507da18", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3e9f66a4-13c7-4391-964b-707e2507da18", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:53Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3e9f66a4-13c7-4391-964b-707e2507da18", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "31f595c7-5674-4e30-bb5b-e742c1577493", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "31f595c7-5674-4e30-bb5b-e742c1577493", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:53Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "31f595c7-5674-4e30-bb5b-e742c1577493", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "dffc7094-fe4b-489e-a9f4-9126e93dc5c2", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "dffc7094-fe4b-489e-a9f4-9126e93dc5c2", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:53Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "dffc7094-fe4b-489e-a9f4-9126e93dc5c2", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "6015f947-25e6-4f7a-9abb-add54b579ab8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "6015f947-25e6-4f7a-9abb-add54b579ab8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:53Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "6015f947-25e6-4f7a-9abb-add54b579ab8", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cdaaecdf-7b43-48cf-81ef-ed7bc7909bf7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cdaaecdf-7b43-48cf-81ef-ed7bc7909bf7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:53Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cdaaecdf-7b43-48cf-81ef-ed7bc7909bf7", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "74611f3a-88db-407b-aea3-958ca43fac9a", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "74611f3a-88db-407b-aea3-958ca43fac9a", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "f7cf7682-5387-42b9-91ad-f16e4a58413f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "f7cf7682-5387-42b9-91ad-f16e4a58413f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4120acc5-c980-458d-b40a-dd2f98ad1d7a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4120acc5-c980-458d-b40a-dd2f98ad1d7a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:06:55Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "openshift-service-ca.crt", "reconcileID": "97c28f3f-3e19-4e2b-87de-6a173f720507", "ConfigMap": "openshift-service-ca.crt", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:55Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "odh-trusted-ca-bundle", "reconcileID": "3e32ab3a-5482-41c8-9b4f-149422f35a7d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "opt-125m-standard-cpu", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:55Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "odh-trusted-ca-bundle", "reconcileID": "3e32ab3a-5482-41c8-9b4f-149422f35a7d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:55Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "openshift-service-ca.crt", "reconcileID": "d58a98ce-ad73-4d1f-a990-4511d7617868", "ConfigMap": "openshift-service-ca.crt", "namespace": "opt-125m-standard-cpu", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:56Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "openshift-service-ca.crt", "reconcileID": "d58a98ce-ad73-4d1f-a990-4511d7617868", "ConfigMap": "openshift-service-ca.crt", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e78c67ce-75f9-4d08-aa3f-090596c72dbf", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "opt-125m-standard-cpu", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:56Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e78c67ce-75f9-4d08-aa3f-090596c72dbf", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "openshift-service-ca.crt", "reconcileID": "8bf516e4-e5e5-484b-b92c-6ac7b8b34d3d", "ConfigMap": "openshift-service-ca.crt", "namespace": "opt-125m-standard-cpu", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:56Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "vllm-runtime", "Namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO Reconcile event triggered by serving runtime: vllm-runtime 2026-07-12T09:06:56Z INFO Multi Node reconciling. {"ResourceName": "vllm-runtime", "Namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO No InferenceServices found for Serving Runtime: vllm-runtime 2026-07-12T09:06:56Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "openshift-service-ca.crt", "reconcileID": "8bf516e4-e5e5-484b-b92c-6ac7b8b34d3d", "ConfigMap": "openshift-service-ca.crt", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "odh-trusted-ca-bundle", "reconcileID": "91aee075-5e93-4621-96c7-b7fcee98c000", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "opt-125m-standard-cpu", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:06:56Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "odh-trusted-ca-bundle", "reconcileID": "91aee075-5e93-4621-96c7-b7fcee98c000", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6ed75f6b-05ff-4675-a44a-13e313207ca9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "opt-125m-standard-cpu", "isvc": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "opt-125m-standard-cpu", "isvc": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:06:56Z INFO inferenceservice-resource Namespace is not protected {"namespace": "opt-125m-standard-cpu", "isvc": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z ERROR Failed to fetch service for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "InferenceService": "opt-125m-standard-cpu"} github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing /workspace/internal/controller/serving/inferenceservice_controller.go:164 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/inferenceservice_controller.go:173 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:06:56Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "create": "opt-125m-standard-cpu-metrics"} 2026-07-12T09:06:56Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "create": "opt-125m-standard-cpu-metrics"} 2026-07-12T09:06:56Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "create": "opt-125m-standard-cpu-metrics-dashboard"} 2026-07-12T09:06:56Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:56Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:56Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:56Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:56Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-standard-cpu {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "9f36f6c4-0378-4cc8-9c57-88a7adafde2d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:56Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "create": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "ConfigMap:": "opt-125m-standard-cpu-metrics-dashboard"} 2026-07-12T09:06:57Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:57Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-standard-cpu {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "5120f827-865a-415e-bfa5-5137893005b1", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "update": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "ConfigMap:": "opt-125m-standard-cpu-metrics-dashboard"} 2026-07-12T09:06:57Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:57Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-standard-cpu {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "a7db9cd8-77b9-4afc-ba04-56fa4da2b39a", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "update": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "ConfigMap:": "opt-125m-standard-cpu-metrics-dashboard"} 2026-07-12T09:06:57Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:57Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:57Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-standard-cpu {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "d4418c5d-a9b7-4f36-aad4-f753a4fdc7b7", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "update": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "ConfigMap:": "opt-125m-standard-cpu-metrics-dashboard"} 2026-07-12T09:06:59Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:59Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:59Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:06:59Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:06:59Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:06:59Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "Namespace": "rhoai-model-registries"} 2026-07-12T09:06:59Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-standard-cpu {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6e64f173-a645-40c2-971d-9ca2dad1b1b4", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "update": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "ConfigMap:": "opt-125m-standard-cpu-metrics-dashboard"} 2026-07-12T09:07:02Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:07:02Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:07:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:07:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:07:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:07:02Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:07:02Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:07:02Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "Namespace": "rhoai-model-registries"} 2026-07-12T09:07:02Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-standard-cpu {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "6c8f7885-03a3-483f-9595-37a377a1747c", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:02Z INFO Multi Node reconciled successfully. {"ResourceName": "vllm-runtime", "Namespace": "opt-125m-standard-cpu"} 2026-07-12T09:07:37Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:07:37Z INFO Multi Node reconciling. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:07:39Z INFO ray ca tls secret modified so ray tls cert need to be updated to sync ca.crt {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:07:39Z INFO Multi Node reconciled successfully. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:17:32Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-4a1dd36b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-4a1dd36b", "reconcileID": "e2368040-83f6-4b7c-abcc-8a299db337ba", "Gateway": "e2e-aigw-4a1dd36b", "namespace": "openshift-ingress"} 2026-07-12T09:17:32Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-4a1dd36b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-4a1dd36b", "reconcileID": "63cb9988-297a-401f-8364-861695e553e9", "Gateway": "e2e-aigw-4a1dd36b", "namespace": "openshift-ingress"} 2026-07-12T09:17:32Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-4a1dd36b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-4a1dd36b", "reconcileID": "5d711730-d297-4df9-af19-8f94909c5dc0", "Gateway": "e2e-aigw-4a1dd36b", "namespace": "openshift-ingress"} 2026-07-12T09:17:32Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-4a1dd36b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-4a1dd36b", "reconcileID": "8e40a62b-6e21-4b0b-bd7b-b11855a7e348", "Gateway": "e2e-aigw-4a1dd36b", "namespace": "openshift-ingress"} 2026-07-12T09:17:32Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "odh-trusted-ca-bundle", "reconcileID": "dcb05532-8e37-4c38-a141-c1872ad38d8d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:17:32Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "odh-trusted-ca-bundle", "reconcileID": "dcb05532-8e37-4c38-a141-c1872ad38d8d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:17:32Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6f620c43-e8ad-49c3-9216-7eff708487ff", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:17:32Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "openshift-service-ca.crt", "reconcileID": "1d4069a7-703b-4211-a9dd-022ff969e366", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:17:32Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "openshift-service-ca.crt", "reconcileID": "d87693bc-72a3-4dcf-8fae-41fcfc50eaea", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:17:32Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "openshift-service-ca.crt", "reconcileID": "d87693bc-72a3-4dcf-8fae-41fcfc50eaea", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:17:32Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "odh-trusted-ca-bundle", "reconcileID": "d323745b-2850-4aad-9d0f-5566dd1c6c56", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:17:32Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "odh-trusted-ca-bundle", "reconcileID": "d323745b-2850-4aad-9d0f-5566dd1c6c56", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:17:32Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "595497c8-17b7-449b-9a99-5c3faee8d489", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:17:32Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "openshift-service-ca.crt", "reconcileID": "af95f961-f319-442a-b1ed-5e98c132cc98", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:17:32Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-4a1dd36b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-4a1dd36b", "reconcileID": "156352e7-1a43-48a6-8ca7-a8befc03994b", "Gateway": "e2e-aigw-4a1dd36b", "namespace": "openshift-ingress"} 2026-07-12T09:17:32Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:17:32Z INFO Multi Node reconciling. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:17:33Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "openshift-service-ca.crt", "reconcileID": "4b1b0189-e637-4df9-989c-5c552220b4b6", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-applications"} 2026-07-12T09:17:34Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-4a1dd36b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-4a1dd36b", "reconcileID": "3139b540-50c7-46d1-8145-d8fb90d8c174", "Gateway": "e2e-aigw-4a1dd36b", "namespace": "openshift-ingress"} 2026-07-12T09:17:34Z INFO ray ca tls secret modified so ray tls cert need to be updated to sync ca.crt {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:17:34Z INFO Multi Node reconciled successfully. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:17:36Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-4a1dd36b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-4a1dd36b", "reconcileID": "1656e5c2-e961-422e-a7ca-cbc808b006ca", "Gateway": "e2e-aigw-4a1dd36b", "namespace": "openshift-ingress"} 2026-07-12T09:17:38Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-cf933f2a","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-cf933f2a", "reconcileID": "811a119e-8c55-4210-a188-fd943775b533", "Gateway": "e2e-aigw-cf933f2a", "namespace": "openshift-ingress"} 2026-07-12T09:17:38Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-cf933f2a","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-cf933f2a", "reconcileID": "de41310a-7cd0-4ce1-b331-e381afed2ba7", "Gateway": "e2e-aigw-cf933f2a", "namespace": "openshift-ingress"} 2026-07-12T09:17:38Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-cf933f2a","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-cf933f2a", "reconcileID": "6a1fdbc6-e959-4927-8a41-13c21b84f2c5", "Gateway": "e2e-aigw-cf933f2a", "namespace": "openshift-ingress"} 2026-07-12T09:17:38Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-cf933f2a","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-cf933f2a", "reconcileID": "10e9fe01-c70b-4a08-b47b-464c904ebe35", "Gateway": "e2e-aigw-cf933f2a", "namespace": "openshift-ingress"} 2026-07-12T09:17:38Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "openshift-service-ca.crt", "reconcileID": "a898289d-6d38-41e7-987d-86c2206890be", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:17:38Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "openshift-service-ca.crt", "reconcileID": "7062c5c7-0250-4fa7-9bfd-af49e34cb289", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-cf933f2a", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:17:38Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "openshift-service-ca.crt", "reconcileID": "7062c5c7-0250-4fa7-9bfd-af49e34cb289", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:17:38Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fbb82893-01f0-4573-958d-5d3fac963d29", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-cf933f2a", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:17:38Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fbb82893-01f0-4573-958d-5d3fac963d29", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:17:38Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "odh-trusted-ca-bundle", "reconcileID": "071769b8-0710-4df6-ad86-cb47a09508f0", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:17:38Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "openshift-service-ca.crt", "reconcileID": "1dccb601-8209-4f01-9a76-fbab46541497", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:17:38Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "07286ad8-1c04-4d19-b868-1f3d8804c280", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:17:38Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "odh-trusted-ca-bundle", "reconcileID": "fe2204ed-a8a9-461d-989c-deb9811363bf", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-cf933f2a", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:17:38Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "odh-trusted-ca-bundle", "reconcileID": "fe2204ed-a8a9-461d-989c-deb9811363bf", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:17:38Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "497cf3ef-b2c9-4815-aad8-e82c74b7f5a6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:17:38Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-cf933f2a","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-cf933f2a", "reconcileID": "883763b3-dbdf-4949-86b0-1938c7b1e60d", "Gateway": "e2e-aigw-cf933f2a", "namespace": "openshift-ingress"} 2026-07-12T09:17:39Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "openshift-service-ca.crt", "reconcileID": "10f0cb7f-e914-42d6-a6a4-f9aaf518d0f6", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-applications"} 2026-07-12T09:17:39Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-cf933f2a","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-cf933f2a", "reconcileID": "1515b287-34e4-42db-bf46-758f5b6d7a30", "Gateway": "e2e-aigw-cf933f2a", "namespace": "openshift-ingress"} 2026-07-12T09:17:40Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-cf933f2a","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-cf933f2a", "reconcileID": "33e6007f-ccfc-455e-b1f7-2bad7de2a1db", "Gateway": "e2e-aigw-cf933f2a", "namespace": "openshift-ingress"} ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-7c7d9467f5-pmns6.log ==================== 2026/07/12 08:53:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46944 - 200 55B in 38.6µs 2026/07/12 08:53:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58394 - 200 55B in 37.587µs 2026/07/12 08:53:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58414 - 200 55B in 36.912µs 2026/07/12 08:53:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55390 - 200 55B in 21.34µs 2026/07/12 08:53:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55416 - 200 55B in 37.256µs 2026/07/12 08:53:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41142 - 200 55B in 38.014µs 2026/07/12 08:54:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41152 - 200 55B in 40.686µs 2026/07/12 08:54:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33328 - 200 55B in 36.968µs 2026/07/12 08:54:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33344 - 200 55B in 61.027µs 2026/07/12 08:54:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39070 - 200 55B in 39.189µs 2026/07/12 08:54:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39096 - 200 55B in 46.454µs 2026/07/12 08:54:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38754 - 200 55B in 38.357µs 2026/07/12 08:54:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38768 - 200 55B in 40.359µs 2026/07/12 08:54:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46496 - 200 55B in 46.611µs 2026/07/12 08:54:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46516 - 200 55B in 37.448µs 2026/07/12 08:54:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43652 - 200 55B in 38.002µs 2026/07/12 08:54:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43662 - 200 55B in 38.391µs 2026/07/12 08:54:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59616 - 200 55B in 37.039µs 2026/07/12 08:55:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59644 - 200 55B in 47.139µs 2026/07/12 08:55:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53844 - 200 55B in 46.967µs 2026/07/12 08:55:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53864 - 200 55B in 36.499µs 2026/07/12 08:55:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42382 - 200 55B in 52.808µs 2026/07/12 08:55:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42394 - 200 55B in 29.591µs 2026/07/12 08:55:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37718 - 200 55B in 37.7µs 2026/07/12 08:55:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37736 - 200 55B in 37.923µs 2026/07/12 08:55:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51882 - 200 55B in 38.32µs 2026/07/12 08:55:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51894 - 200 55B in 38.096µs 2026/07/12 08:55:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40126 - 200 55B in 45.72µs 2026/07/12 08:55:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40142 - 200 55B in 38.705µs 2026/07/12 08:55:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34728 - 200 55B in 21.608µs 2026/07/12 08:56:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34736 - 200 55B in 38.814µs 2026/07/12 08:56:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37848 - 200 55B in 42.713µs 2026/07/12 08:56:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37868 - 200 55B in 38.336µs 2026/07/12 08:56:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52496 - 200 55B in 37.076µs 2026/07/12 08:56:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52508 - 200 55B in 38.963µs 2026/07/12 08:56:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50148 - 200 55B in 43.305µs 2026/07/12 08:56:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50168 - 200 55B in 39.389µs 2026/07/12 08:56:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44510 - 200 55B in 58.71µs 2026/07/12 08:56:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44516 - 200 55B in 38.766µs 2026/07/12 08:56:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56868 - 200 55B in 44.099µs 2026/07/12 08:56:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56890 - 200 55B in 37.937µs 2026/07/12 08:56:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59228 - 200 55B in 37.769µs 2026/07/12 08:57:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59258 - 200 55B in 65.599µs 2026/07/12 08:57:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54804 - 200 55B in 39.778µs 2026/07/12 08:57:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54820 - 200 55B in 38.694µs 2026/07/12 08:57:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40514 - 200 55B in 38.953µs 2026/07/12 08:57:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40524 - 200 55B in 23.738µs 2026/07/12 08:57:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42088 - 200 55B in 38.429µs 2026/07/12 08:57:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42116 - 200 55B in 62.396µs 2026/07/12 08:57:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33570 - 200 55B in 58.159µs 2026/07/12 08:57:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33592 - 200 55B in 38.204µs 2026/07/12 08:57:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46240 - 200 55B in 39.303µs 2026/07/12 08:57:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46262 - 200 55B in 38.937µs 2026/07/12 08:57:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52096 - 200 55B in 38.252µs 2026/07/12 08:58:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52112 - 200 55B in 21.853µs 2026/07/12 08:58:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50134 - 200 55B in 37.417µs 2026/07/12 08:58:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50152 - 200 55B in 37.281µs 2026/07/12 08:58:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53244 - 200 55B in 39.771µs 2026/07/12 08:58:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53268 - 200 55B in 40.77µs 2026/07/12 08:58:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42948 - 200 55B in 219.642µs 2026/07/12 08:58:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42966 - 200 55B in 37.305µs 2026/07/12 08:58:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48890 - 200 55B in 40.587µs 2026/07/12 08:58:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48908 - 200 55B in 38.247µs 2026/07/12 08:58:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55820 - 200 55B in 44.541µs 2026/07/12 08:58:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55834 - 200 55B in 37.541µs 2026/07/12 08:58:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48662 - 200 55B in 43.139µs 2026/07/12 08:59:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48674 - 200 55B in 38.588µs 2026/07/12 08:59:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41724 - 200 55B in 41.516µs 2026/07/12 08:59:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41734 - 200 55B in 36.815µs 2026/07/12 08:59:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54600 - 200 55B in 49.116µs 2026/07/12 08:59:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54614 - 200 55B in 38.445µs 2026/07/12 08:59:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34826 - 200 55B in 39.667µs 2026/07/12 08:59:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34854 - 200 55B in 23.362µs 2026/07/12 08:59:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50880 - 200 55B in 37.371µs 2026/07/12 08:59:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50906 - 200 55B in 38.715µs 2026/07/12 08:59:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45458 - 200 55B in 40.888µs 2026/07/12 08:59:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45466 - 200 55B in 36.999µs 2026/07/12 08:59:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55300 - 200 55B in 39.934µs 2026/07/12 09:00:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55318 - 200 55B in 51.289µs 2026/07/12 09:00:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53614 - 200 55B in 38.542µs 2026/07/12 09:00:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53632 - 200 55B in 75.277µs 2026/07/12 09:00:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36214 - 200 55B in 41.212µs 2026/07/12 09:00:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36242 - 200 55B in 39.396µs 2026/07/12 09:00:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58428 - 200 55B in 52.776µs 2026/07/12 09:00:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58456 - 200 55B in 38.27µs 2026/07/12 09:00:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36930 - 200 55B in 42.076µs 2026/07/12 09:00:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36952 - 200 55B in 45.564µs 2026/07/12 09:00:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58002 - 200 55B in 37.572µs 2026/07/12 09:00:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58022 - 200 55B in 47.328µs 2026/07/12 09:00:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52894 - 200 55B in 47.466µs 2026/07/12 09:01:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52918 - 200 55B in 41.711µs 2026/07/12 09:01:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45996 - 200 55B in 38.949µs 2026/07/12 09:01:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46016 - 200 55B in 37.11µs 2026/07/12 09:01:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44946 - 200 55B in 50.34µs 2026/07/12 09:01:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44968 - 200 55B in 22.385µs 2026/07/12 09:01:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52868 - 200 55B in 37.313µs 2026/07/12 09:01:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52880 - 200 55B in 37.948µs 2026/07/12 09:01:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51204 - 200 55B in 21.935µs 2026/07/12 09:01:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51220 - 200 55B in 43.049µs 2026/07/12 09:01:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40258 - 200 55B in 48.166µs 2026/07/12 09:01:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40264 - 200 55B in 38.447µs 2026/07/12 09:01:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44784 - 200 55B in 37.986µs 2026/07/12 09:02:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44798 - 200 55B in 44.368µs 2026/07/12 09:02:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53924 - 200 55B in 36.843µs 2026/07/12 09:02:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53950 - 200 55B in 41.121µs 2026/07/12 09:02:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39970 - 200 55B in 40.953µs 2026/07/12 09:02:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39994 - 200 55B in 38.749µs 2026/07/12 09:02:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58210 - 200 55B in 39.482µs 2026/07/12 09:02:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58244 - 200 55B in 38.947µs 2026/07/12 09:02:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48268 - 200 55B in 65.684µs 2026/07/12 09:02:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48284 - 200 55B in 37.647µs 2026/07/12 09:02:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46546 - 200 55B in 42.099µs 2026/07/12 09:02:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46568 - 200 55B in 36.807µs 2026/07/12 09:02:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60394 - 200 55B in 38.727µs 2026/07/12 09:03:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60398 - 200 55B in 38.386µs 2026/07/12 09:03:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58606 - 200 55B in 41.752µs 2026/07/12 09:03:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58634 - 200 55B in 36.581µs 2026/07/12 09:03:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42102 - 200 55B in 27.645µs 2026/07/12 09:03:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42126 - 200 55B in 37.674µs 2026/07/12 09:03:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59354 - 200 55B in 38.567µs 2026/07/12 09:03:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59376 - 200 55B in 21.618µs 2026/07/12 09:03:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57608 - 200 55B in 21.962µs 2026/07/12 09:03:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57624 - 200 55B in 44.604µs 2026/07/12 09:03:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59242 - 200 55B in 37.022µs 2026/07/12 09:03:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59254 - 200 55B in 37.538µs 2026/07/12 09:03:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41230 - 200 55B in 48.405µs 2026/07/12 09:04:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41254 - 200 55B in 45.187µs 2026/07/12 09:04:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34520 - 200 55B in 41.49µs 2026/07/12 09:04:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34542 - 200 55B in 38.813µs 2026/07/12 09:04:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57998 - 200 55B in 56.864µs 2026/07/12 09:04:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58030 - 200 55B in 40.273µs 2026/07/12 09:04:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53624 - 200 55B in 38.016µs 2026/07/12 09:04:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53640 - 200 55B in 39.598µs 2026/07/12 09:04:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39724 - 200 55B in 38.391µs 2026/07/12 09:04:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39752 - 200 55B in 58.584µs 2026/07/12 09:04:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38236 - 200 55B in 43.441µs 2026/07/12 09:04:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38260 - 200 55B in 38.043µs 2026/07/12 09:04:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41438 - 200 55B in 40.588µs 2026/07/12 09:05:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41452 - 200 55B in 46.631µs 2026/07/12 09:05:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43536 - 200 55B in 57.365µs 2026/07/12 09:05:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43560 - 200 55B in 49.345µs 2026/07/12 09:05:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43098 - 200 55B in 23.636µs 2026/07/12 09:05:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43104 - 200 55B in 23.318µs 2026/07/12 09:05:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33486 - 200 55B in 41.916µs 2026/07/12 09:05:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33512 - 200 55B in 58.008µs 2026/07/12 09:05:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38370 - 200 55B in 48.854µs 2026/07/12 09:05:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38384 - 200 55B in 38.669µs 2026/07/12 09:05:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58084 - 200 55B in 28.008µs 2026/07/12 09:05:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58104 - 200 55B in 40.357µs 2026/07/12 09:05:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44638 - 200 55B in 39.403µs 2026/07/12 09:06:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44648 - 200 55B in 39.595µs 2026/07/12 09:06:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56982 - 200 55B in 38.509µs 2026/07/12 09:06:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57000 - 200 55B in 36.156µs 2026/07/12 09:06:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41958 - 200 55B in 70.172µs 2026/07/12 09:06:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41978 - 200 55B in 63.022µs 2026/07/12 09:06:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44950 - 200 55B in 37.59µs 2026/07/12 09:06:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44960 - 200 55B in 55.873µs 2026/07/12 09:06:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53728 - 200 55B in 73.823µs 2026/07/12 09:06:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53750 - 200 55B in 37.423µs 2026/07/12 09:06:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45820 - 200 55B in 53.08µs 2026/07/12 09:06:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45844 - 200 55B in 36.837µs 2026/07/12 09:06:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54490 - 200 55B in 53.452µs 2026/07/12 09:07:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54512 - 200 55B in 47.46µs 2026/07/12 09:07:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56906 - 200 55B in 39.501µs 2026/07/12 09:07:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56926 - 200 55B in 60.439µs 2026/07/12 09:07:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54974 - 200 55B in 42.21µs 2026/07/12 09:07:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55000 - 200 55B in 22.709µs 2026/07/12 09:07:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55246 - 200 55B in 37.8µs 2026/07/12 09:07:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55270 - 200 55B in 21.181µs 2026/07/12 09:07:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55908 - 200 55B in 42.877µs 2026/07/12 09:07:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55926 - 200 55B in 39.333µs 2026/07/12 09:07:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37176 - 200 55B in 47.906µs 2026/07/12 09:07:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37196 - 200 55B in 39.008µs 2026/07/12 09:07:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50926 - 200 55B in 47.591µs 2026/07/12 09:08:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50936 - 200 55B in 43.304µs 2026/07/12 09:08:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58522 - 200 55B in 54.777µs 2026/07/12 09:08:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58534 - 200 55B in 39.121µs 2026/07/12 09:08:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46356 - 200 55B in 37.54µs 2026/07/12 09:08:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46376 - 200 55B in 38.236µs 2026/07/12 09:08:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48184 - 200 55B in 39.434µs 2026/07/12 09:08:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48206 - 200 55B in 38.351µs 2026/07/12 09:08:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33536 - 200 55B in 45.433µs 2026/07/12 09:08:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33562 - 200 55B in 44.063µs 2026/07/12 09:08:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45216 - 200 55B in 46.072µs 2026/07/12 09:08:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45226 - 200 55B in 40.72µs 2026/07/12 09:08:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59022 - 200 55B in 49.463µs 2026/07/12 09:09:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59038 - 200 55B in 39.535µs 2026/07/12 09:09:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47650 - 200 55B in 50.646µs 2026/07/12 09:09:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47676 - 200 55B in 37.159µs 2026/07/12 09:09:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38464 - 200 55B in 28.233µs 2026/07/12 09:09:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38488 - 200 55B in 22.235µs 2026/07/12 09:09:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40988 - 200 55B in 38.863µs 2026/07/12 09:09:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41012 - 200 55B in 38.37µs 2026/07/12 09:09:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58236 - 200 55B in 37.497µs 2026/07/12 09:09:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58258 - 200 55B in 39.856µs 2026/07/12 09:09:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52906 - 200 55B in 37.063µs 2026/07/12 09:09:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52920 - 200 55B in 23.097µs 2026/07/12 09:09:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46876 - 200 55B in 37.796µs 2026/07/12 09:10:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46882 - 200 55B in 46.995µs 2026/07/12 09:10:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59490 - 200 55B in 55.974µs 2026/07/12 09:10:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59496 - 200 55B in 37.998µs 2026/07/12 09:10:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46634 - 200 55B in 45.185µs 2026/07/12 09:10:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46638 - 200 55B in 43.442µs 2026/07/12 09:10:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38908 - 200 55B in 39.041µs 2026/07/12 09:10:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38924 - 200 55B in 40.611µs 2026/07/12 09:10:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43796 - 200 55B in 38.78µs 2026/07/12 09:10:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43822 - 200 55B in 37.01µs 2026/07/12 09:10:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42166 - 200 55B in 49.175µs 2026/07/12 09:10:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42182 - 200 55B in 46.923µs 2026/07/12 09:10:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57064 - 200 55B in 35.215µs 2026/07/12 09:11:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57074 - 200 55B in 40.812µs 2026/07/12 09:11:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47980 - 200 55B in 40.22µs 2026/07/12 09:11:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47994 - 200 55B in 40.818µs 2026/07/12 09:11:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48868 - 200 55B in 40.895µs 2026/07/12 09:11:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48888 - 200 55B in 40.193µs 2026/07/12 09:11:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48242 - 200 55B in 37.701µs 2026/07/12 09:11:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48262 - 200 55B in 42.116µs 2026/07/12 09:11:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58946 - 200 55B in 39.103µs 2026/07/12 09:11:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58964 - 200 55B in 38.497µs 2026/07/12 09:11:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34674 - 200 55B in 39.124µs 2026/07/12 09:11:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34694 - 200 55B in 37.451µs 2026/07/12 09:11:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45044 - 200 55B in 58.26µs 2026/07/12 09:12:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45056 - 200 55B in 39.146µs 2026/07/12 09:12:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33172 - 200 55B in 39.346µs 2026/07/12 09:12:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33180 - 200 55B in 23.217µs 2026/07/12 09:12:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39456 - 200 55B in 39.408µs 2026/07/12 09:12:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39480 - 200 55B in 38.095µs 2026/07/12 09:12:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50874 - 200 55B in 38.409µs 2026/07/12 09:12:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50894 - 200 55B in 37.221µs 2026/07/12 09:12:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36616 - 200 55B in 38.007µs 2026/07/12 09:12:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36624 - 200 55B in 38.433µs 2026/07/12 09:12:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41648 - 200 55B in 54.501µs 2026/07/12 09:12:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41656 - 200 55B in 37.762µs 2026/07/12 09:12:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44822 - 200 55B in 39.627µs 2026/07/12 09:13:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44838 - 200 55B in 37.404µs 2026/07/12 09:13:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36436 - 200 55B in 36.852µs 2026/07/12 09:13:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36444 - 200 55B in 38.886µs 2026/07/12 09:13:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55950 - 200 55B in 37.609µs 2026/07/12 09:13:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55966 - 200 55B in 40.826µs 2026/07/12 09:13:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:32976 - 200 55B in 23.076µs 2026/07/12 09:13:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:32984 - 200 55B in 35.812µs 2026/07/12 09:13:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47670 - 200 55B in 39.761µs 2026/07/12 09:13:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47688 - 200 55B in 73.629µs 2026/07/12 09:13:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40852 - 200 55B in 51.768µs 2026/07/12 09:13:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40864 - 200 55B in 34.246µs 2026/07/12 09:13:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50772 - 200 55B in 37.796µs 2026/07/12 09:14:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50782 - 200 55B in 39.363µs 2026/07/12 09:14:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59364 - 200 55B in 40.987µs 2026/07/12 09:14:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59374 - 200 55B in 38.475µs 2026/07/12 09:14:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35530 - 200 55B in 45.74µs 2026/07/12 09:14:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35552 - 200 55B in 39.326µs 2026/07/12 09:14:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42952 - 200 55B in 44.183µs 2026/07/12 09:14:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42968 - 200 55B in 43.392µs 2026/07/12 09:14:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36040 - 200 55B in 45.416µs 2026/07/12 09:14:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36054 - 200 55B in 41.25µs 2026/07/12 09:14:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40142 - 200 55B in 46.226µs 2026/07/12 09:14:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40156 - 200 55B in 39.505µs 2026/07/12 09:14:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60914 - 200 55B in 35.931µs 2026/07/12 09:15:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60942 - 200 55B in 46.507µs 2026/07/12 09:15:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54498 - 200 55B in 40.338µs 2026/07/12 09:15:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54510 - 200 55B in 39.432µs 2026/07/12 09:15:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57762 - 200 55B in 41.187µs 2026/07/12 09:15:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57774 - 200 55B in 39.246µs 2026/07/12 09:15:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46694 - 200 55B in 37.05µs 2026/07/12 09:15:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46724 - 200 55B in 46.159µs 2026/07/12 09:15:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51520 - 200 55B in 52.34µs 2026/07/12 09:15:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51538 - 200 55B in 44.798µs 2026/07/12 09:15:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43308 - 200 55B in 38.192µs 2026/07/12 09:15:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43332 - 200 55B in 39.339µs 2026/07/12 09:15:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38908 - 200 55B in 47.779µs 2026/07/12 09:16:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38920 - 200 55B in 38.432µs 2026/07/12 09:16:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59424 - 200 55B in 38.227µs 2026/07/12 09:16:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59448 - 200 55B in 37.518µs 2026/07/12 09:16:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39490 - 200 55B in 47.152µs 2026/07/12 09:16:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39498 - 200 55B in 38.317µs 2026/07/12 09:16:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44262 - 200 55B in 41.233µs 2026/07/12 09:16:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44302 - 200 55B in 43.032µs 2026/07/12 09:16:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54764 - 200 55B in 39.02µs 2026/07/12 09:16:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54784 - 200 55B in 39.562µs 2026/07/12 09:16:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58618 - 200 55B in 68.929µs 2026/07/12 09:16:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58638 - 200 55B in 37.239µs 2026/07/12 09:16:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49336 - 200 55B in 38.399µs 2026/07/12 09:17:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49346 - 200 55B in 75.447µs 2026/07/12 09:17:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39836 - 200 55B in 39.103µs 2026/07/12 09:17:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39856 - 200 55B in 38.266µs 2026/07/12 09:17:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46172 - 200 55B in 39.274µs 2026/07/12 09:17:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46176 - 200 55B in 40.789µs 2026/07/12 09:17:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59252 - 200 55B in 37.832µs 2026/07/12 09:17:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59272 - 200 55B in 38.882µs 2026/07/12 09:17:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50988 - 200 55B in 86.241µs 2026/07/12 09:17:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51006 - 200 55B in 49.433µs 2026/07/12 09:17:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34766 - 200 55B in 47.613µs 2026/07/12 09:17:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34780 - 200 55B in 38.362µs 2026/07/12 09:17:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57692 - 200 55B in 36.987µs 2026/07/12 09:18:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57712 - 200 55B in 45.892µs 2026/07/12 09:18:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48436 - 200 55B in 49.774µs 2026/07/12 09:18:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48462 - 200 55B in 48.7µs 2026/07/12 09:18:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55150 - 200 55B in 59.775µs ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-566f8bdddc-66b9q.log ====================