==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== 121m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 134m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 134m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 45m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 45m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 170m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 170m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 134m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 134m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 170m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 170m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning FailedMount pod/maas-api-e2e-aigw-cf933f2a-5f6d987c7b-kh96d MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-aigw-cf933f2a-serving-cert" not found 2m25s Warning TooManyMissedTimes cronjob/maas-api-key-cleanup too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew 34s Warning FailedMount pod/maas-api-e2e-aigw-ba61e35b-654d467d46-krhwp MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-aigw-ba61e35b-serving-cert" not found 31s Warning FailedMount pod/maas-api-e2e-aigw-ef12e752-596dcd6c8-2p6q4 MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-aigw-ef12e752-serving-cert" not found {"level":"error","ts":"2026-07-12T09:37:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:38:48Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:39:23Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:40:08Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:40:56Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:41:39Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:42:11Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:43:02Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:43:34Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:44:13Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:45:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:45:36Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:46:25Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:47:24Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:48:20Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:49:06Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:49:55Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:50:28Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:51:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:51:54Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:52:42Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:53:30Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:54:07Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:54:52Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:55:33Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:56:21Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:57:15Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:58:10Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:58:48Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:59:30Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:00:15Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:01:06Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:01:54Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:02:38Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:03:36Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:04:07Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-07-12T10:04:14Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.19 Kubernetes Version: v1.34.8 --- Namespaces --- namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.1 Authorino Operator 1.4.1 authorino-operator.v1.3.2 Succeeded cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Succeeded custom-metrics-autoscaler.v2.19.0-1 Custom Metrics Autoscaler 2.19.0-1 custom-metrics-autoscaler.v2.18.1-2 Succeeded dns-operator.v1.4.0 DNS Operator 1.4.0 dns-operator.v1.3.1 Succeeded kueue-operator.v1.3.1 Red Hat build of Kueue 1.3.1 Succeeded limitador-operator.v1.4.0 Limitador Operator 1.4.0 limitador-operator.v1.3.1 Succeeded rhcl-operator.v1.4.1 Red Hat Connectivity Link 1.4.1 rhcl-operator.v1.3.5 Succeeded rhods-operator.3.5.0-ea.2 Red Hat OpenShift AI 3.5.0-ea.2 Succeeded servicemeshoperator3.v3.2.0 Red Hat OpenShift Service Mesh 3 3.2.0 servicemeshoperator3.v3.1.10 Pending servicemeshoperator3.v3.4.0 Red Hat OpenShift Service Mesh 3 3.4.0 servicemeshoperator3.v3.2.0 Pending --- CSVs not in Succeeded phase --- servicemeshoperator3.v3.2.0: phase=Pending reason=Copied message=The operator is running in openshift-operators but is managing this namespace servicemeshoperator3.v3.4.0: phase=Pending reason=Copied message=The operator is running in openshift-operators but is managing this namespace --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aigateway: managementState=n/a aipipelines: managementState=Managed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Unmanaged llamastackoperator: managementState=n/a mlflowoperator: managementState=Managed modelregistry: managementState=Managed ogx: managementState=Removed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Managed trustyai: managementState=Managed workbenches: managementState=Managed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= AIPipelinesReady: status=True reason= message= DashboardReady: status=True reason= message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=True reason= message= LLM-D-WVADependencies: status=True reason= message= MLflowOperatorReady: status=True reason= message= MaaSPrerequisitesAvailable: status=True reason=MaaSPrerequisitesMet message=MaaS gateway annotations and Authorino TLS are correctly configured ModelControllerReady: status=True reason= message= ModelRegistryReady: status=True reason= message= ModelsAsServiceReady: status=True reason=Reconciled message=MaaS platform manifests applied and maas-api deployment is available OGXReady: status=False reason=Removed message=Component ManagementState is set to Removed RayReady: status=True reason= message= SparkOperatorReady: status=True reason= message= TrainerReady: status=True reason= message= TrainingOperatorReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason= message= --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 2d3h Ready 2026-07-10T06:04:24Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in DSCI CR OpenTelemetryCollectorAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Metrics not configured in DSCI CR Traces not configured in DSCI CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in DSCI CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR PersesAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR --- Component CRs (Ready) --- NAME READY REASON URL dashboard.components.platform.opendatahub.io/default-dashboard True NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON kueue.components.platform.opendatahub.io/default-kueue True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True NAME READY REASON URI modelcontroller.components.platform.opendatahub.io/default-modelcontroller True NAME READY REASON modelregistry.components.platform.opendatahub.io/default-modelregistry True NAME READY REASON modelsasservice.components.platform.opendatahub.io/default-modelsasservice True NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True NAME READY REASON trainer.components.platform.opendatahub.io/default-trainer True NAME READY REASON trainingoperator.components.platform.opendatahub.io/default-trainingoperator True NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON workbenches.components.platform.opendatahub.io/default-workbenches True --- Deployments not fully ready (ODH/RHOAI namespaces) --- (all deployments report ready replicas == desired) --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- (all pods Running or Succeeded) --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 121m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 134m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 134m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 45m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 45m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 170m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 170m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 121m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 134m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 134m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 157m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 170m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 170m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning FailedMount pod/maas-api-e2e-aigw-cf933f2a-5f6d987c7b-kh96d MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-aigw-cf933f2a-serving-cert" not found 2m25s Warning TooManyMissedTimes cronjob/maas-api-key-cleanup too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew 34s Warning FailedMount pod/maas-api-e2e-aigw-ba61e35b-654d467d46-krhwp MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-aigw-ba61e35b-serving-cert" not found 31s Warning FailedMount pod/maas-api-e2e-aigw-ef12e752-596dcd6c8-2p6q4 MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-e2e-aigw-ef12e752-serving-cert" not found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- (no events) --- namespace/rhoai-model-registries --- (no events) --- namespace/rhods-notebooks --- (no events) --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-07-12T09:37:44Z) --- redhat-ods-operator/rhods-operator-6d59b54db7-vnrh2 --- {"level":"error","ts":"2026-07-12T09:37:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:38:48Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:39:23Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:40:08Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:40:56Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:41:39Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:42:11Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:43:02Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:43:34Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:44:13Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:45:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:45:36Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:46:25Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:47:24Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:48:20Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:49:06Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:49:55Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:50:28Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:51:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:51:54Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:52:42Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:53:30Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:54:07Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:54:52Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:55:33Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:56:21Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:57:15Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:58:10Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:58:48Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:59:30Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:00:15Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:01:06Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:01:54Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:02:38Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:03:36Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:04:07Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-6d59b54db7-qs29r.log ==================== ==================== OPERATOR POD LOG: rhods-operator-6d59b54db7-rgvz5.log ==================== ==================== OPERATOR POD LOG: rhods-operator-6d59b54db7-vnrh2.log ==================== {"level":"info","ts":"2026-07-12T09:37:44Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:37:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:38:48Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:39:04Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:39:23Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:39:57Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:40:08Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:40:56Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:41:39Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:42:11Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:43:02Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:43:32Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b86f82f-15e0-4305-8fd0-c9fe8667c72f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:43:33Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"61e747c9-0f88-4e11-896c-12e9b197a837","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:43:33Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"61e747c9-0f88-4e11-896c-12e9b197a837","name":"redhat-ods-applications"} {"level":"error","ts":"2026-07-12T09:43:34Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:44:13Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:45:04Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:45:36Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:45:52Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-12T09:45:54Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-kserve-raw-token-authentication","reconcileID":"8bc57bf5-1245-4ac5-8bc2-7d0a53ab3b9c"} {"level":"info","ts":"2026-07-12T09:45:54Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-kserve-raw-token-authentication","reconcileID":"02225b00-db69-4d93-a216-38c8ad75ddd3"} {"level":"info","ts":"2026-07-12T09:45:54Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-kserve-raw-token-authentication","reconcileID":"9c89478a-dfc9-4851-80ac-2e3fe6448fc0"} {"level":"error","ts":"2026-07-12T09:46:25Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:46:39Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-12T09:47:05Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-non-admin-raw","reconcileID":"965e7980-cfe3-4df3-b363-582714cc5767"} {"level":"info","ts":"2026-07-12T09:47:05Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-non-admin-raw","reconcileID":"fe22f03a-fe0a-4b64-8a8e-ca11845c8182"} {"level":"info","ts":"2026-07-12T09:47:05Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-non-admin-raw","reconcileID":"0bf00c09-775a-460d-a684-2ac0e9793c67"} {"level":"error","ts":"2026-07-12T09:47:24Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:48:06Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"sklearn-modelcar","reconcileID":"d7652bbf-4f9e-4e64-9d99-dd346d980d25"} {"level":"info","ts":"2026-07-12T09:48:06Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"sklearn-modelcar","reconcileID":"34f61bea-c51c-4d93-b81e-e1cd8e87ac70"} {"level":"info","ts":"2026-07-12T09:48:06Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"sklearn-modelcar","reconcileID":"65535dbd-85d7-43cc-92fa-ab69a7cd64da"} {"level":"error","ts":"2026-07-12T09:48:20Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:48:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"onnx-standard-rest","reconcileID":"9f14b1e4-6c1e-4cf4-b67b-8195a3050805"} {"level":"info","ts":"2026-07-12T09:48:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"onnx-standard-rest","reconcileID":"e8c8b01f-e36c-4c4f-a5fd-3387d2894232"} {"level":"info","ts":"2026-07-12T09:48:29Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"onnx-standard-rest","reconcileID":"6927ae44-f055-4f8d-ae61-9958a1fc9216"} {"level":"info","ts":"2026-07-12T09:48:53Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ovms-smoke","reconcileID":"55425166-b7ea-47c9-bb19-b02e79f03125"} {"level":"info","ts":"2026-07-12T09:48:53Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ovms-smoke","reconcileID":"154e98db-5abe-4963-a8bf-2cb7eb321304"} {"level":"info","ts":"2026-07-12T09:48:53Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ovms-smoke","reconcileID":"908a3dba-37a2-48be-9488-318ecac419c5"} {"level":"error","ts":"2026-07-12T09:49:06Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:49:24Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-12T09:49:32Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"tensorflow-standard","reconcileID":"2eeddf59-4d70-4f27-83a5-a752435a266e"} {"level":"info","ts":"2026-07-12T09:49:32Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"tensorflow-standard","reconcileID":"85ec26c8-0e0b-4c1b-bfe6-92516d648814"} {"level":"info","ts":"2026-07-12T09:49:32Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"tensorflow-standard","reconcileID":"5b42d32b-f947-4d92-94da-056d404a9999"} {"level":"error","ts":"2026-07-12T09:49:55Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:50:08Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"opt-125m-probes","reconcileID":"85965e81-5d72-4842-a431-c525cb846ed2"} {"level":"info","ts":"2026-07-12T09:50:08Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"opt-125m-probes","reconcileID":"3309b25c-0c85-407d-bdb7-be27a431085c"} {"level":"info","ts":"2026-07-12T09:50:08Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"opt-125m-probes","reconcileID":"6964876b-c3ce-4898-bb8c-6454ff982017"} {"level":"error","ts":"2026-07-12T09:50:28Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:51:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:51:54Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:52:02Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:52:42Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:53:30Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:54:07Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2db04155-3cd1-4209-9bee-0589e8ca2ac5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T09:54:15Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"46fc0648-c08b-4892-8185-7e22f99a93d5","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T09:54:15Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"46fc0648-c08b-4892-8185-7e22f99a93d5","name":"redhat-ods-applications"} {"level":"error","ts":"2026-07-12T09:54:52Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:55:33Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:55:39Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:56:21Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:57:15Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T09:57:16Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T09:58:10Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:58:48Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T09:59:30Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:00:15Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T10:01:03Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T10:01:06Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:01:54Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T10:01:57Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-12T10:02:38Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-12T10:03:36Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-ba61e35b","reconcileID":"927cbcb4-c81c-492b-b421-8586bcb01f8c"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-ba61e35b","reconcileID":"99296bcf-5490-43b6-a612-5bb65c1ed225"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"437845e4-a738-48e0-81ef-0ba33298087b"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"437845e4-a738-48e0-81ef-0ba33298087b"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"437845e4-a738-48e0-81ef-0ba33298087b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"437845e4-a738-48e0-81ef-0ba33298087b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"437845e4-a738-48e0-81ef-0ba33298087b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"437845e4-a738-48e0-81ef-0ba33298087b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"437845e4-a738-48e0-81ef-0ba33298087b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-ba61e35b","reconcileID":"54b0dfb7-ff51-4e66-82e0-e122fad712b0"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"437845e4-a738-48e0-81ef-0ba33298087b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"437845e4-a738-48e0-81ef-0ba33298087b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"09970b84-28b7-4e1f-8428-acb860cab46e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"f370659a-735e-4fd6-80e9-cdd17dee1de6","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"f370659a-735e-4fd6-80e9-cdd17dee1de6","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fc54394-a652-4956-b2e7-e45ecd8aea7b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"813743a7-a054-4189-b24a-684bbbc80f20"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"813743a7-a054-4189-b24a-684bbbc80f20"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"813743a7-a054-4189-b24a-684bbbc80f20","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"813743a7-a054-4189-b24a-684bbbc80f20","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"813743a7-a054-4189-b24a-684bbbc80f20","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"813743a7-a054-4189-b24a-684bbbc80f20","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"813743a7-a054-4189-b24a-684bbbc80f20","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"813743a7-a054-4189-b24a-684bbbc80f20","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"813743a7-a054-4189-b24a-684bbbc80f20","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0a56c7dc-58a5-406a-a165-8dace05e49e6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"4cae3a16-b061-4ca9-a8f6-77756bfdf9d5","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"4cae3a16-b061-4ca9-a8f6-77756bfdf9d5","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"9dbde44f-92f5-4001-9be4-e9356923a6fe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"65c6d31a-2223-45bd-b6b7-6eba32783d64","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:43Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2b243a01-b701-4cc3-b95f-48bb6d39946d","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2b243a01-b701-4cc3-b95f-48bb6d39946d","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1ef17441-f4b1-492c-8042-bf4d252364bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b01df490-c9f9-4664-8c86-2a350ace5f4c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"92be8faf-dca8-4f25-a2f1-c87fb5992a48","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"92be8faf-dca8-4f25-a2f1-c87fb5992a48","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c78778c8-bd70-4494-90f1-180b7f5d0a65","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"31f31de7-b3f7-45fc-8d1f-7e5b26fb2a6b"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"31f31de7-b3f7-45fc-8d1f-7e5b26fb2a6b"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"31f31de7-b3f7-45fc-8d1f-7e5b26fb2a6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"31f31de7-b3f7-45fc-8d1f-7e5b26fb2a6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"31f31de7-b3f7-45fc-8d1f-7e5b26fb2a6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"31f31de7-b3f7-45fc-8d1f-7e5b26fb2a6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"31f31de7-b3f7-45fc-8d1f-7e5b26fb2a6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"31f31de7-b3f7-45fc-8d1f-7e5b26fb2a6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"31f31de7-b3f7-45fc-8d1f-7e5b26fb2a6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29bc8a92-8b46-4742-8ede-f06cc1d68472"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29bc8a92-8b46-4742-8ede-f06cc1d68472"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29bc8a92-8b46-4742-8ede-f06cc1d68472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29bc8a92-8b46-4742-8ede-f06cc1d68472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29bc8a92-8b46-4742-8ede-f06cc1d68472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29bc8a92-8b46-4742-8ede-f06cc1d68472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29bc8a92-8b46-4742-8ede-f06cc1d68472","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29bc8a92-8b46-4742-8ede-f06cc1d68472","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"29bc8a92-8b46-4742-8ede-f06cc1d68472","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"21deec57-62ba-4426-ae28-2a4687ab0db1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"18e3ba70-06d2-4840-9561-2cd8cf414f80","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"18e3ba70-06d2-4840-9561-2cd8cf414f80","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b9223329-ab55-4adc-bf18-f565f051c6b8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-ef12e752","reconcileID":"11714b8d-342f-43fe-aa5c-39aa79a3e661"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"653266a9-c62a-4bb4-a733-6ab4efbae8fb"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"653266a9-c62a-4bb4-a733-6ab4efbae8fb"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"653266a9-c62a-4bb4-a733-6ab4efbae8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"653266a9-c62a-4bb4-a733-6ab4efbae8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"653266a9-c62a-4bb4-a733-6ab4efbae8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"653266a9-c62a-4bb4-a733-6ab4efbae8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"653266a9-c62a-4bb4-a733-6ab4efbae8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"653266a9-c62a-4bb4-a733-6ab4efbae8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-ef12e752","reconcileID":"f71b8afc-4715-43f4-8d75-92eb5831251f"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"ai-tenant-e2e-aigw-ef12e752","reconcileID":"9f9c674d-637e-429a-9ba3-19e9de32a351"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"653266a9-c62a-4bb4-a733-6ab4efbae8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a1fc3fa6-f9d9-452c-b956-abb2c568eea8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-12T10:03:46Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"669ae780-8d56-44da-a1c9-b196d007ade0","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"669ae780-8d56-44da-a1c9-b196d007ade0","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"69e67082-2594-4173-b769-5dcf61df40a6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8786f4-a302-443f-b647-6cd8e76e2a21"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8786f4-a302-443f-b647-6cd8e76e2a21"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8786f4-a302-443f-b647-6cd8e76e2a21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8786f4-a302-443f-b647-6cd8e76e2a21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8786f4-a302-443f-b647-6cd8e76e2a21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8786f4-a302-443f-b647-6cd8e76e2a21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8786f4-a302-443f-b647-6cd8e76e2a21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8786f4-a302-443f-b647-6cd8e76e2a21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8786f4-a302-443f-b647-6cd8e76e2a21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3a8839ff-357e-42f6-8902-1dbdefe8a8ac"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3a8839ff-357e-42f6-8902-1dbdefe8a8ac"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3a8839ff-357e-42f6-8902-1dbdefe8a8ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3a8839ff-357e-42f6-8902-1dbdefe8a8ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3a8839ff-357e-42f6-8902-1dbdefe8a8ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3a8839ff-357e-42f6-8902-1dbdefe8a8ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3a8839ff-357e-42f6-8902-1dbdefe8a8ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3a8839ff-357e-42f6-8902-1dbdefe8a8ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3a8839ff-357e-42f6-8902-1dbdefe8a8ac","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5a43deae-2c42-4870-9166-6d0621de2103","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"9c7fb21c-34e2-46b3-a2ea-e5dc48eaac64","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"9c7fb21c-34e2-46b3-a2ea-e5dc48eaac64","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d562f888-0089-4d8e-925e-f37e1dc89952","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c53f26dd-778a-4ae0-8367-c137ffcb20f6","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ddbe5908-aed4-4d46-8a30-8bf21ec73010","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ddbe5908-aed4-4d46-8a30-8bf21ec73010","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-12T10:03:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"8767b5f9-f519-4493-934a-a1ab29840b01","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-12T10:04:07Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-84fcf4d86-2j5cr.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf7988tz9x2.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-clkg6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-controller-manager-dd9f6cf9-l5vmt.log ==================== {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa"}} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:54Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"cd7cc535-baf5-4a7c-9149-4b33f6d83350"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"error","ts":"2026-07-12T09:45:54Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"cd7cc535-baf5-4a7c-9149-4b33f6d83350","error":"fails to reconcile ingress: Route.route.openshift.io \"http-onnx\" not found","errorVerbose":"Route.route.openshift.io \"http-onnx\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:54Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"error","ts":"2026-07-12T09:45:55Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"656556ec-1710-4b66-b198-fcd591f9ab99","error":"fails to reconcile ingress: Route.route.openshift.io \"http-onnx\" not found","errorVerbose":"Route.route.openshift.io \"http-onnx\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"error","ts":"2026-07-12T09:45:55Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"df141ba1-2fe5-44bf-8006-5cf751447d44","error":"fails to reconcile ingress: Route.route.openshift.io \"http-onnx\" not found","errorVerbose":"Route.route.openshift.io \"http-onnx\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"error","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:45:55Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"66d98cc1-354e-47b6-84bf-4b1ce8e1fffe","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:55Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:56Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:45:58Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"DeploymentReconciler","msg":"Deployment Updated","Diff":" v1.DeploymentSpec{\n \t... // 1 ignored field\n \tSelector: &{MatchLabels: {\"app\": \"isvc.http-onnx-predictor\"}},\n \tTemplate: v1.PodTemplateSpec{\n \t\tObjectMeta: v1.ObjectMeta{\n \t\t\t... // 9 identical fields\n \t\t\tDeletionGracePeriodSeconds: nil,\n \t\t\tLabels: {\"app\": \"isvc.http-onnx-predictor\", \"component\": \"predictor\", \"networking.kserve.io/visibility\": \"exposed\", \"serving.kserve.io/inferenceservice\": \"http-onnx\"},\n \t\t\tAnnotations: map[string]string{\n \t\t\t\t... // 4 identical entries\n \t\t\t\t\"prometheus.io/path\": \"/metrics\",\n \t\t\t\t\"prometheus.io/port\": \"8888\",\n- \t\t\t\t\"security.opendatahub.io/enable-auth\": \"true\",\n+ \t\t\t\t\"security.opendatahub.io/enable-auth\": \"false\",\n \t\t\t\t\"service.beta.openshift.io/serving-cert-secret-name\": \"http-onnx-predictor-serving-cert\",\n \t\t\t\t\"serving.kserve.io/deploymentMode\": \"Standard\",\n \t\t\t},\n \t\t\tOwnerReferences: nil,\n \t\t\tFinalizers: nil,\n \t\t\tManagedFields: nil,\n \t\t},\n \t\tSpec: {Volumes: {{Name: \"proxy-tls\", VolumeSource: {Secret: &{SecretName: \"http-onnx-predictor-serving-cert\", DefaultMode: &420}}}, {Name: \"kube-rbac-proxy-sar-config\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"http-onnx-kube-rbac-proxy-sar-config\"}, DefaultMode: &420}}}}, Containers: {{Name: \"kserve-container\", Image: \"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:\"..., Args: {\"--model_name=http-onnx\", \"--port=8001\", \"--rest_port=8888\", \"--model_path=/mnt/models\", ...}, Ports: {{ContainerPort: 8888, Protocol: \"TCP\"}}, ...}, {Name: \"kube-rbac-proxy\", Image: \"registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e\"..., Args: {\"--secure-listen-address=:8443\", \"--proxy-endpoints-port=8643\", \"--upstream=http://localhost:8888\", \"--auth-header-fields-enabled=true\", ...}, Ports: {{Name: \"https\", ContainerPort: 8443, Protocol: \"TCP\"}, {Name: \"proxy\", ContainerPort: 8643, Protocol: \"TCP\"}}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &30, ...},\n \t},\n \tStrategy: {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"25%\"}, MaxSurge: &{Type: 1, StrVal: \"25%\"}}},\n \tMinReadySeconds: 0,\n \t... // 3 identical fields\n }\n"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"error","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:46:17Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"416167f9-875c-4613-892d-c127db5df1e6","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:20Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"DeploymentReconciler","msg":"Deployment Updated","Diff":" v1.DeploymentSpec{\n \t... // 1 ignored field\n \tSelector: &{MatchLabels: {\"app\": \"isvc.http-onnx-predictor\"}},\n \tTemplate: v1.PodTemplateSpec{\n \t\tObjectMeta: v1.ObjectMeta{\n \t\t\t... // 9 identical fields\n \t\t\tDeletionGracePeriodSeconds: nil,\n \t\t\tLabels: {\"app\": \"isvc.http-onnx-predictor\", \"component\": \"predictor\", \"networking.kserve.io/visibility\": \"exposed\", \"serving.kserve.io/inferenceservice\": \"http-onnx\"},\n \t\t\tAnnotations: map[string]string{\n \t\t\t\t... // 4 identical entries\n \t\t\t\t\"prometheus.io/path\": \"/metrics\",\n \t\t\t\t\"prometheus.io/port\": \"8888\",\n- \t\t\t\t\"security.opendatahub.io/enable-auth\": \"false\",\n+ \t\t\t\t\"security.opendatahub.io/enable-auth\": \"true\",\n \t\t\t\t\"service.beta.openshift.io/serving-cert-secret-name\": \"http-onnx-predictor-serving-cert\",\n \t\t\t\t\"serving.kserve.io/deploymentMode\": \"Standard\",\n \t\t\t},\n \t\t\tOwnerReferences: nil,\n \t\t\tFinalizers: nil,\n \t\t\tManagedFields: nil,\n \t\t},\n \t\tSpec: {Volumes: {{Name: \"proxy-tls\", VolumeSource: {Secret: &{SecretName: \"http-onnx-predictor-serving-cert\", DefaultMode: &420}}}, {Name: \"kube-rbac-proxy-sar-config\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"http-onnx-kube-rbac-proxy-sar-config\"}, DefaultMode: &420}}}}, Containers: {{Name: \"kserve-container\", Image: \"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:\"..., Args: {\"--model_name=http-onnx\", \"--port=8001\", \"--rest_port=8888\", \"--model_path=/mnt/models\", ...}, Ports: {{ContainerPort: 8888, Protocol: \"TCP\"}}, ...}, {Name: \"kube-rbac-proxy\", Image: \"registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e\"..., Args: {\"--secure-listen-address=:8443\", \"--proxy-endpoints-port=8643\", \"--upstream=http://localhost:8888\", \"--auth-header-fields-enabled=true\", ...}, Ports: {{Name: \"https\", ContainerPort: 8443, Protocol: \"TCP\"}, {Name: \"proxy\", ContainerPort: 8643, Protocol: \"TCP\"}}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &30, ...},\n \t},\n \tStrategy: {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"25%\"}, MaxSurge: &{Type: 1, StrVal: \"25%\"}}},\n \tMinReadySeconds: 0,\n \t... // 3 identical fields\n }\n"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"error","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:46:24Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx","reconcileID":"d0f1e170-4bf3-444f-a7e2-3715965cea82","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx"}],"resources":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:24Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2"}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx-2","reconcileID":"42afbc63-2629-4ee7-a789-6812f96de017"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"error","ts":"2026-07-12T09:46:31Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx-2","reconcileID":"42afbc63-2629-4ee7-a789-6812f96de017","error":"fails to reconcile ingress: Route.route.openshift.io \"http-onnx-2\" not found","errorVerbose":"Route.route.openshift.io \"http-onnx-2\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"error","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx-2","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-2\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Error updating status after ingress reconcile failure","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-2\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-2\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:381\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:46:31Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"},"namespace":"test-kserve-raw-token-authentication","name":"http-onnx-2","reconcileID":"00406051-64c7-4b9d-a41b-4c04e8dc991a","error":"fails to reconcile ingress: Route.route.openshift.io \"http-onnx-2\" not found","errorVerbose":"Route.route.openshift.io \"http-onnx-2\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:31Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:34Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-kserve-raw-token-authentication"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-2","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-2"}],"resources":{}}],"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"http-onnx-2"} {"level":"info","ts":"2026-07-12T09:46:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:53Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:53Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa-2","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"http-onnx"} {"level":"info","ts":"2026-07-12T09:46:54Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:54Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:46:54Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:46:54Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-kserve-raw-token-authentication","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-secret"}},"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}}}} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:05Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-raw","namespace":"test-non-admin-raw"},"namespace":"test-non-admin-raw","name":"http-onnx-raw","reconcileID":"7d7f4908-230c-4499-90a2-bfada033a546"} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:05Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"error","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx-raw","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-raw\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:47:06Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-raw","namespace":"test-non-admin-raw"},"namespace":"test-non-admin-raw","name":"http-onnx-raw","reconcileID":"6ed6ed3e-2276-4b3c-9b86-a644b6e636f9","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-raw\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-raw\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"error","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"http-onnx-raw","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-raw\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:47:06Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"http-onnx-raw","namespace":"test-non-admin-raw"},"namespace":"test-non-admin-raw","name":"http-onnx-raw","reconcileID":"5285bdac-77f2-44d8-ba13-f6a566d30357","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-raw\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"http-onnx-raw\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:06Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"http-onnx-raw","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"test-non-admin-raw"} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=http-onnx-raw","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"http-onnx-raw"}],"resources":{}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:47:16Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"http-onnx-raw"} {"level":"info","ts":"2026-07-12T09:47:18Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:47:18Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:47:18Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"test-non-admin-raw","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"http-mnist-runtime","name":"","resources":{},"storage":{"path":"/test-dir/","key":"ci-bucket-unprivileged"}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}}}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"sklearn-modelcar","namespace":"sklearn-modelcar"},"namespace":"sklearn-modelcar","name":"sklearn-modelcar","reconcileID":"b7c7fd68-9d92-42c4-aef5-3bba01edd167"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"error","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"sklearn-modelcar","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"sklearn-modelcar\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:48:07Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"sklearn-modelcar","namespace":"sklearn-modelcar"},"namespace":"sklearn-modelcar","name":"sklearn-modelcar","reconcileID":"5020cbda-da6f-4f07-a149-e63d32e2a441","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"sklearn-modelcar\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"sklearn-modelcar\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"error","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"sklearn-modelcar","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"sklearn-modelcar\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:48:07Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"sklearn-modelcar","namespace":"sklearn-modelcar"},"namespace":"sklearn-modelcar","name":"sklearn-modelcar","reconcileID":"33a82b90-9015-49e8-9a27-1af9d2b6d940","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"sklearn-modelcar\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"sklearn-modelcar\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:07Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:08Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:08Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:08Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:08Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:08Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:09Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:09Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:09Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:09Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:09Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:17Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:17Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:17Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"sklearn-modelcar","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:bd383f9ea096c157352344018911847d7bfcc53a380091605edb1f474625e2af","ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"MLSERVER_MODEL_NAME","value":"sklearn-modelcar"},{"name":"MLSERVER_MODEL_URI","value":"/mnt/models"},{"name":"MLSERVER_MODEL_IMPLEMENTATION","value":"mlserver_sklearn.SKLearnModel"},{"name":"MLSERVER_HTTP_PORT","value":"8080"},{"name":"MLSERVER_MODELS_DIR","value":"/mnt/models"},{"name":"INFERENCE_SERVICE_NAME","value":"sklearn-modelcar"}],"resources":{},"livenessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":20,"timeoutSeconds":5,"periodSeconds":10,"failureThreshold":6},"readinessProbe":{"httpGet":{"path":"/v2/models/sklearn-modelcar/ready","port":8080},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":5,"failureThreshold":3},"startupProbe":{"exec":{"command":["/bin/sh","-c","[ -n \"$(ls -A /mnt/models 2>/dev/null)\" ]\n"]},"initialDelaySeconds":1,"periodSeconds":1,"failureThreshold":1},"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true,"allowPrivilegeEscalation":false}}],"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"error","ts":"2026-07-12T09:48:18Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"sklearn-modelcar","namespace":"sklearn-modelcar"},"namespace":"sklearn-modelcar","name":"sklearn-modelcar","reconcileID":"e8ed319d-5b9e-49b2-a76f-dd907a8ca481","error":"InferenceService.serving.kserve.io \"sklearn-modelcar\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"odh.inferenceservice.finalizers\"}","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"sklearn-modelcar"} {"level":"info","ts":"2026-07-12T09:48:18Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"sklearn-modelcar","isvc":{"model":{"modelFormat":{"name":"sklearn"},"runtime":"mlserver-runtime","storageUri":"oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55","name":"","resources":{}},"automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:19Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:19Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"sklearn-modelcar"} {"level":"error","ts":"2026-07-12T09:48:19Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"sklearn-modelcar","namespace":"sklearn-modelcar"},"namespace":"sklearn-modelcar","name":"sklearn-modelcar","reconcileID":"e66ce044-e7e6-487f-b270-82a674a25840","error":"inferenceservices.serving.kserve.io \"sklearn-modelcar\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"onnx"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"onnx"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"onnx"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"onnx"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"onnx","namespace":"onnx-standard-rest"},"namespace":"onnx-standard-rest","name":"onnx","reconcileID":"e7d6fd14-f054-4cf6-8c96-932a083a4693"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"onnx"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:48:29Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"CredentialBuilder","msg":"Setting secret envs for s3","S3Secret":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"onnx"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"openvino-models-bucket-sa-dockercfg-nmgjr"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"error","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"onnx","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"onnx\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:48:30Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"onnx","namespace":"onnx-standard-rest"},"namespace":"onnx-standard-rest","name":"onnx","reconcileID":"987a59d5-c242-47b9-915a-7426a850cc64","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"onnx\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"onnx\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:30Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:32Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"onnx","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"onnx-standard-rest"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8","args":["--model_name=onnx","--port=8001","--rest_port=8888","--model_path=/mnt/models","--file_system_poll_wait_seconds=0","--metrics_enable"],"ports":[{"containerPort":8888,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"onnx"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"onnx"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"onnx"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:48:41Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"onnx-standard-rest","isvc":{"model":{"modelFormat":{"name":"openvino_ir"},"runtime":"openvino-runtime","storageUri":"s3://mlflow-e2e/openvino/model_repository/onnx/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"openvino-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:32Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"},"namespace":"tensorflow-standard","name":"tensorflow-standard-rest","reconcileID":"e191458d-73a4-4545-b374-f34a67429ba0"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CredentialBuilder","msg":"Setting secret envs for s3","S3Secret":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"triton-models-bucket-sa-dockercfg-9gdsw"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"error","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"tensorflow-standard-rest","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"tensorflow-standard-rest\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:49:33Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"},"namespace":"tensorflow-standard","name":"tensorflow-standard-rest","reconcileID":"51158cde-f98f-4992-8079-00ca37116eb3","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"tensorflow-standard-rest\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"tensorflow-standard-rest\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:402\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:33Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:35Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:37Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"tensorflow-standard-rest","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"tensorflow-standard"} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"containers":[{"name":"kserve-container","image":"nvcr.io/nvidia/tritonserver:25.02-py3","args":["tritonserver","--model-store=/mnt/models","--http-port=8080","--allow-http=True"],"ports":[{"name":"http1","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"INFERENCE_SERVICE_NAME","value":"tensorflow-standard-rest"}],"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}],"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:49:46Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Deleting external resources","InferenceService":"tensorflow-standard-rest"} {"level":"info","ts":"2026-07-12T09:49:57Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:57Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:49:57Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:49:57Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"tensorflow-standard","isvc":{"model":{"modelFormat":{"name":"tensorrt"},"runtime":"triton-rest-runtime","storageUri":"s3://mlflow-e2e/triton/model_repository/","name":"","resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"serviceAccountName":"triton-models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-probes","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate create","name":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-probes","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-probes","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-probes","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:08Z","msg":"metadata.finalizers: \"inferenceservice.finalizers\": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"opt-125m-probes","namespace":"opt-125m-probes"},"namespace":"opt-125m-probes","name":"opt-125m-probes","reconcileID":"45b2ed09-6966-42a5-8276-995001680d41"} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-probes","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-probes"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}],"livenessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":180,"timeoutSeconds":10,"periodSeconds":30,"failureThreshold":10},"readinessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":120,"timeoutSeconds":10,"periodSeconds":10,"failureThreshold":12}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":0,"err":null} {"level":"info","ts":"2026-07-12T09:50:08Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-probes"} {"level":"error","ts":"2026-07-12T09:50:08Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"opt-125m-probes","namespace":"opt-125m-probes"},"namespace":"opt-125m-probes","name":"opt-125m-probes","reconcileID":"45b2ed09-6966-42a5-8276-995001680d41","error":"fails to reconcile ingress: Route.route.openshift.io \"opt-125m-probes\" not found","errorVerbose":"Route.route.openshift.io \"opt-125m-probes\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-probes","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-probes"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}],"livenessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":180,"timeoutSeconds":10,"periodSeconds":30,"failureThreshold":10},"readinessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":120,"timeoutSeconds":10,"periodSeconds":10,"failureThreshold":12}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"opt-125m-probes","isvc":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"CredentialBuilder","msg":"Setting secret envs for s3","S3Secret":"models-bucket-secret"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"validate update","name":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"inferenceservice-v1beta1-validation-webhook","msg":"Incoming predictor struct","predictor":{"model":{"modelFormat":{"name":"vLLM"},"runtime":"vllm-runtime","storageUri":"s3://mlflow-e2e/opt-125m/","protocolVersion":"v2","name":"","args":["--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}]},"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false,"minReplicas":1}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"CredentialBuilder","msg":"found secret","SecretName":"models-bucket-sa-dockercfg-pr2xd"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-probes"} {"level":"error","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Failed to update InferenceService status","InferenceService":"opt-125m-probes","error":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"opt-125m-probes\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:428\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Error updating status after ingress reconcile failure","error":"fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"opt-125m-probes\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on inferenceservices.serving.kserve.io \"opt-125m-probes\": the object has been modified; please apply your changes to the latest version and try again\nfails to update InferenceService status\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).updateStatus\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:431\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:380\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:381\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"error","ts":"2026-07-12T09:50:09Z","msg":"Reconciler error","controller":"inferenceservice","controllerGroup":"serving.kserve.io","controllerKind":"InferenceService","InferenceService":{"name":"opt-125m-probes","namespace":"opt-125m-probes"},"namespace":"opt-125m-probes","name":"opt-125m-probes","reconcileID":"3aad282b-8062-4075-9f79-02b8f04ac821","error":"fails to reconcile ingress: Route.route.openshift.io \"opt-125m-probes\" not found","errorVerbose":"Route.route.openshift.io \"opt-125m-probes\" not found\nfails to reconcile ingress\ngithub.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\t/go/src/github.com/kserve/kserve/pkg/controller/v1beta1/inferenceservice/controller.go:383\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1693","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-probes","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-probes"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}],"livenessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":180,"timeoutSeconds":10,"periodSeconds":30,"failureThreshold":10},"readinessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":120,"timeoutSeconds":10,"periodSeconds":10,"failureThreshold":12}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"kserve-pod-mutator-webhook","msg":"The Inference Service does not contain a logging spec. The global ConfigMap will be used as the logger configuration","name":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"kserve-pod-mutator-webhook","msg":"getLoggerConfig processing configuration","loggerConfig":{"image":"registry.redhat.io/rhoai/odh-kserve-agent-rhel9@sha256:22624819998af78cce29b57772463b810d1a881b8f9a76d3c74e224c77da004e","cpuRequest":"100m","cpuLimit":"1","memoryRequest":"100Mi","memoryLimit":"1Gi","defaultUrl":"http://default-broker","caBundle":"","caCertFile":"","tlsSkipVerify":false,"storage":null}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-probes","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-probes"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}],"livenessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":180,"timeoutSeconds":10,"periodSeconds":30,"failureThreshold":10},"readinessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":120,"timeoutSeconds":10,"periodSeconds":10,"failureThreshold":12}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-probes","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-probes"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}],"livenessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":180,"timeoutSeconds":10,"periodSeconds":30,"failureThreshold":10},"readinessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":120,"timeoutSeconds":10,"periodSeconds":10,"failureThreshold":12}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-probes","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-probes"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}],"livenessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":180,"timeoutSeconds":10,"periodSeconds":30,"failureThreshold":10},"readinessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":120,"timeoutSeconds":10,"periodSeconds":10,"failureThreshold":12}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-probes","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-probes"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}],"livenessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":180,"timeoutSeconds":10,"periodSeconds":30,"failureThreshold":10},"readinessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":120,"timeoutSeconds":10,"periodSeconds":10,"failureThreshold":12}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T09:50:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T10:00:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Inference service deployment mode ","deployment mode ":"Standard"} {"level":"info","ts":"2026-07-12T10:00:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling inference service","apiVersion":"serving.kserve.io/v1beta1","isvc":"opt-125m-probes","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T10:00:09Z","logger":"CaBundleConfigMapReconciler","msg":"Reconciling CaBundleConfigMap","namespace":"opt-125m-probes"} {"level":"info","ts":"2026-07-12T10:00:09Z","logger":"PredictorReconciler","msg":"Resolved container","container":"nil","podSpec":{"volumes":[{"name":"shared-memory","emptyDir":{"medium":"Memory","sizeLimit":"32Gi"}},{"name":"tmp","emptyDir":{}},{"name":"home","emptyDir":{}}],"containers":[{"name":"kserve-container","image":"registry.redhat.io/rhaii-early-access/vllm-cpu-rhel9@sha256:c953143f67160cb6afc3439a788e4f314d31311eacc68c33ff934f69ff023ed7","command":["python","-m","vllm.entrypoints.openai.api_server"],"args":["--port=8080","--model=/mnt/models","--served-model-name=opt-125m-probes","--model=/mnt/models","--uvicorn-log-level=debug","--enforce-eager","--max-model-len=256","--max-num-seqs=20"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"env":[{"name":"VLLM_CPU_KVCACHE_SPACE","value":"4"},{"name":"VLLM_WORKER_MULTIPROC_METHOD","value":"spawn"},{"name":"OMP_NUM_THREADS","value":"8"},{"name":"HF_HOME","value":"/tmp/hf_home"},{"name":"INFERENCE_SERVICE_NAME","value":"opt-125m-probes"}],"resources":{"limits":{"cpu":"16","memory":"16Gi"},"requests":{"cpu":"8","memory":"10Gi"}},"volumeMounts":[{"name":"shared-memory","mountPath":"/dev/shm"},{"name":"tmp","mountPath":"/tmp"},{"name":"home","mountPath":"/home/vllm"}],"livenessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":180,"timeoutSeconds":10,"periodSeconds":30,"failureThreshold":10},"readinessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":120,"timeoutSeconds":10,"periodSeconds":10,"failureThreshold":12}}],"serviceAccountName":"models-bucket-sa","automountServiceAccountToken":false}} {"level":"info","ts":"2026-07-12T10:00:09Z","logger":"ServiceReconciler","msg":"service reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T10:00:09Z","logger":"DeploymentReconciler","msg":"deployment reconcile","checkResult":2,"err":null} {"level":"info","ts":"2026-07-12T10:00:09Z","logger":"HPAReconciler","msg":"HorizontalPodAutoscaler reconcile","checkResult":1,"err":null} {"level":"info","ts":"2026-07-12T10:00:09Z","logger":"v1beta1Controllers.InferenceService","msg":"Reconciling ingress for inference service","isvc":"opt-125m-probes"} ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-kqcfg.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-training-operator-54786944d6-rm4lg.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kuberay-operator-6586cdc875-j9bqf.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-56w7f.log ==================== {"level":"info","ts":"2026-07-12T09:43:23Z","logger":"controller-runtime.cache","msg":"Warning: VariantAutoscaling is deprecated and will be removed in a future release. Migrate to the annotation-based path (add llm-d.ai/managed=true to your HPA or ScaledObject). See docs/developer-guide/migrating-from-va-crd.md for migration steps."} {"level":"info","ts":"2026-07-12T09:48:58Z","logger":"controller-runtime.cache","msg":"Warning: VariantAutoscaling is deprecated and will be removed in a future release. Migrate to the annotation-based path (add llm-d.ai/managed=true to your HPA or ScaledObject). See docs/developer-guide/migrating-from-va-crd.md for migration steps."} {"level":"info","ts":"2026-07-12T09:58:00Z","logger":"controller-runtime.cache","msg":"Warning: VariantAutoscaling is deprecated and will be removed in a future release. Migrate to the annotation-based path (add llm-d.ai/managed=true to your HPA or ScaledObject). See docs/developer-guide/migrating-from-va-crd.md for migration steps."} ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-7d4778f7cc-4vfq8.log ==================== {"level":"info","timestamp":"2026-07-12T09:43:34.418Z","caller":"config/cluster_config.go:103","message":"Created MaaSModelRef informer","watchNamespace":"ALL","gvr":"maas.opendatahub.io/v1alpha1, Resource=maasmodelrefs"} {"level":"info","timestamp":"2026-07-12T09:43:34.418Z","caller":"config/cluster_config.go:110","message":"Created MaaSSubscription informer","watchNamespace":"models-as-a-service","gvr":"maas.opendatahub.io/v1alpha1, Resource=maassubscriptions"} {"level":"info","timestamp":"2026-07-12T09:43:34.419Z","caller":"cmd/main.go:66","message":"Loading database connection URL from secret..."} {"level":"info","timestamp":"2026-07-12T09:43:34.436Z","caller":"cmd/main.go:173","message":"Connecting to PostgreSQL database...","tenant":"models-as-a-service"} {"level":"info","timestamp":"2026-07-12T09:43:34.436Z","caller":"cmd/main.go:102","message":"Metrics server starting","address":":9090"} {"level":"info","timestamp":"2026-07-12T09:43:34.519Z","caller":"api_keys/db_driver.go:91","message":"Database schema applied","version":5} {"level":"info","timestamp":"2026-07-12T09:43:34.519Z","caller":"api_keys/db_driver.go:58","message":"Connected to PostgreSQL database (schema applied)","tenant":"models-as-a-service"} {"level":"info","timestamp":"2026-07-12T09:43:34.519Z","caller":"cmd/main.go:180","message":"Starting informers and waiting for cache sync..."} {"level":"info","timestamp":"2026-07-12T09:43:34.620Z","caller":"cmd/main.go:184","message":"Informer caches synced successfully"} {"level":"info","timestamp":"2026-07-12T09:43:34.627Z","caller":"cmd/main.go:202","message":"Resolved gateway internal host for access probes","host":"maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local"} {"level":"info","timestamp":"2026-07-12T09:43:34.638Z","caller":"cmd/main.go:135","message":"Server starting","address":":8443","secure":true} [GIN] 2026/07/12 - 09:43:40 | 200 | 63.561µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:43:45 | 200 | 32.343µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:43:50 | 200 | 29.812µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:43:55 | 200 | 30.55µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:00 | 200 | 33.852µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:05 | 200 | 33.837µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:10 | 200 | 31.809µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:13 | 200 | 43.394µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:15 | 200 | 34.657µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:20 | 200 | 45.402µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:23 | 200 | 36.953µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:25 | 200 | 39.806µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:30 | 200 | 46.466µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:33 | 200 | 32.448µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:35 | 200 | 38.489µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:40 | 200 | 42.084µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:43 | 200 | 38.489µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:45 | 200 | 39.717µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:50 | 200 | 42.373µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:53 | 200 | 40.338µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:44:55 | 200 | 40.635µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:00 | 200 | 40.252µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:03 | 200 | 38.107µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:05 | 200 | 41.409µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:10 | 200 | 42.168µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:13 | 200 | 50.007µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:15 | 200 | 43.416µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:20 | 200 | 42.77µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:23 | 200 | 46.36µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:25 | 200 | 37.974µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:30 | 200 | 42.928µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:33 | 200 | 40.099µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:35 | 200 | 37.972µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:40 | 200 | 39.247µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:43 | 200 | 39.026µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:45 | 200 | 37.77µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:50 | 200 | 41.52µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:53 | 200 | 55.543µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:45:55 | 200 | 32.036µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:00 | 200 | 45.532µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:03 | 200 | 39.407µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:05 | 200 | 32.465µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:10 | 200 | 42.766µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:13 | 200 | 33.601µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:15 | 200 | 39.113µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:20 | 200 | 32.822µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:23 | 200 | 31.094µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:25 | 200 | 33.889µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:30 | 200 | 35.486µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:33 | 200 | 32.733µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:35 | 200 | 37.386µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:40 | 200 | 34.021µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:43 | 200 | 33.956µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:45 | 200 | 34.22µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:50 | 200 | 32.586µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:53 | 200 | 32.849µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:46:55 | 200 | 34.556µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:00 | 200 | 34.592µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:03 | 200 | 36.693µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:05 | 200 | 54.839µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:10 | 200 | 32.777µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:13 | 200 | 33.017µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:15 | 200 | 32.665µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:20 | 200 | 34.951µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:23 | 200 | 32.369µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:25 | 200 | 32.021µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:30 | 200 | 33.174µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:33 | 200 | 41.953µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:35 | 200 | 33.261µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:40 | 200 | 34.364µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:43 | 200 | 31.679µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:45 | 200 | 31.531µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:50 | 200 | 32.869µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:53 | 200 | 35.605µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:47:55 | 200 | 33.693µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:00 | 200 | 29.588µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:03 | 200 | 33.517µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:05 | 200 | 32.534µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:10 | 200 | 36.378µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:13 | 200 | 43.688µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:15 | 200 | 39.76µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:20 | 200 | 37.692µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:23 | 200 | 45.292µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:25 | 200 | 36.477µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:30 | 200 | 38.115µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:33 | 200 | 39.104µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:35 | 200 | 39.912µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:40 | 200 | 39.324µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:43 | 200 | 39.627µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:45 | 200 | 37.887µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:50 | 200 | 37.311µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:53 | 200 | 41.006µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:48:55 | 200 | 42.697µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:00 | 200 | 38.332µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:03 | 200 | 48.76µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:05 | 200 | 53.984µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:10 | 200 | 39.025µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:13 | 200 | 40.699µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:15 | 200 | 68.193µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:20 | 200 | 41.783µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:23 | 200 | 42.81µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:25 | 200 | 42.492µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:30 | 200 | 42.613µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:33 | 200 | 39.385µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:35 | 200 | 58.932µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:40 | 200 | 40.565µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:43 | 200 | 37.314µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:45 | 200 | 39.199µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:50 | 200 | 39.509µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:53 | 200 | 40.22µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:49:55 | 200 | 50.248µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:00 | 200 | 35.716µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:03 | 200 | 40.238µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:05 | 200 | 37.491µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:10 | 200 | 37.959µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:13 | 200 | 38.666µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:15 | 200 | 44.032µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:20 | 200 | 38.672µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:23 | 200 | 42.629µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:25 | 200 | 41.174µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:30 | 200 | 38.765µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:33 | 200 | 39.59µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:35 | 200 | 41.829µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:40 | 200 | 42.301µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:43 | 200 | 42.273µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:45 | 200 | 39.271µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:50 | 200 | 37.235µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:53 | 200 | 37.933µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:50:55 | 200 | 38.01µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:00 | 200 | 38.674µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:03 | 200 | 41.332µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:05 | 200 | 42.141µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:10 | 200 | 40.323µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:13 | 200 | 39.297µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:15 | 200 | 41.104µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:20 | 200 | 36.259µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:23 | 200 | 38.374µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:25 | 200 | 39.278µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:30 | 200 | 40.109µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:33 | 200 | 33.524µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:35 | 200 | 37.955µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:40 | 200 | 43.29µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:43 | 200 | 42.59µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:45 | 200 | 34.004µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:50 | 200 | 40.669µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:53 | 200 | 43.971µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:51:55 | 200 | 31.088µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:00 | 200 | 45.425µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:03 | 200 | 32.818µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:05 | 200 | 34.169µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:10 | 200 | 31.438µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:13 | 200 | 32.22µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:15 | 200 | 31.472µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:20 | 200 | 29.804µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:23 | 200 | 32.708µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:25 | 200 | 32.19µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:30 | 200 | 30.797µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:33 | 200 | 31.51µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:35 | 200 | 31.735µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:40 | 200 | 32.888µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:43 | 200 | 39.028µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:45 | 200 | 31.87µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:50 | 200 | 39.507µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:53 | 200 | 33.77µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:52:55 | 200 | 34.929µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:00 | 200 | 32.337µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:03 | 200 | 34.203µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:05 | 200 | 33.126µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:10 | 200 | 35.635µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:13 | 200 | 34.04µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:15 | 200 | 31.737µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:20 | 200 | 34.904µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:23 | 200 | 33.212µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:25 | 200 | 32.458µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:30 | 200 | 41.323µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:33 | 200 | 34.585µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:35 | 200 | 31.292µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:40 | 200 | 33.381µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:43 | 200 | 32.041µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:45 | 200 | 31.912µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:50 | 200 | 46.545µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:53 | 200 | 30.314µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:53:55 | 200 | 33.009µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:00 | 200 | 30.555µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:03 | 200 | 36.895µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:05 | 200 | 37.14µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:10 | 200 | 36.084µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:13 | 200 | 63.005µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:15 | 200 | 58.196µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:20 | 200 | 57.296µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:23 | 200 | 43.852µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:25 | 200 | 42.949µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:30 | 200 | 38.333µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:33 | 200 | 54.694µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:35 | 200 | 45.161µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:40 | 200 | 41.813µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:43 | 200 | 40.117µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:45 | 200 | 41.727µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:50 | 200 | 37.441µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:53 | 200 | 40.744µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:54:55 | 200 | 40.185µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:00 | 200 | 43.736µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:03 | 200 | 40.07µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:05 | 200 | 39.264µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:10 | 200 | 58.827µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:13 | 200 | 41.872µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:15 | 200 | 37.825µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:20 | 200 | 36.262µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:23 | 200 | 44.612µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:25 | 200 | 40.133µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:30 | 200 | 37.792µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:33 | 200 | 45.596µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:35 | 200 | 41.895µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:40 | 200 | 47.023µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:43 | 200 | 38.553µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:45 | 200 | 42.337µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:50 | 200 | 40.259µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:53 | 200 | 37.689µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:55:55 | 200 | 40.042µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:00 | 200 | 38.697µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:03 | 200 | 37.582µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:05 | 200 | 36.943µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:10 | 200 | 45.091µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:13 | 200 | 38.697µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:15 | 200 | 44.42µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:20 | 200 | 45.155µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:23 | 200 | 38.49µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:25 | 200 | 38.282µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:30 | 200 | 40.882µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:33 | 200 | 38.76µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:35 | 200 | 55.178µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:40 | 200 | 39.965µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:43 | 200 | 40.036µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:45 | 200 | 41.983µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:50 | 200 | 42.067µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:53 | 200 | 37.252µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:56:55 | 200 | 39.849µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:00 | 200 | 38.344µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:03 | 200 | 35.027µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:05 | 200 | 32.248µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:10 | 200 | 36.068µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:13 | 200 | 39.721µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:15 | 200 | 38.105µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:20 | 200 | 38.831µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:23 | 200 | 37.125µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:25 | 200 | 43.027µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:30 | 200 | 41.581µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:33 | 200 | 38.583µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:35 | 200 | 37.864µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:40 | 200 | 39.865µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:43 | 200 | 37.987µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:45 | 200 | 36.524µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:50 | 200 | 39.175µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:53 | 200 | 37.871µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:57:55 | 200 | 38.074µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:00 | 200 | 38.952µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:03 | 200 | 43.385µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:05 | 200 | 33.187µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:10 | 200 | 33.25µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:13 | 200 | 34.405µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:15 | 200 | 32.277µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:20 | 200 | 31.573µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:23 | 200 | 34.136µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:25 | 200 | 31.492µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:30 | 200 | 31.662µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:33 | 200 | 35.082µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:35 | 200 | 32.873µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:40 | 200 | 34.937µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:43 | 200 | 32.492µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:45 | 200 | 34.446µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:50 | 200 | 31.704µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:53 | 200 | 34.918µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:58:55 | 200 | 33.951µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:00 | 200 | 32.739µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:03 | 200 | 31.927µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:05 | 200 | 32.528µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:10 | 200 | 35.743µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:13 | 200 | 30.562µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:15 | 200 | 34.918µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:20 | 200 | 31.703µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:23 | 200 | 32.955µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:25 | 200 | 36.117µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:30 | 200 | 32.864µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:33 | 200 | 34.235µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:35 | 200 | 33.653µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:40 | 200 | 33.087µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:43 | 200 | 35.342µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:45 | 200 | 33.624µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:50 | 200 | 32.77µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:53 | 200 | 30.658µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 09:59:55 | 200 | 35.527µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:00 | 200 | 32.543µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:03 | 200 | 32.513µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:05 | 200 | 32.406µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:10 | 200 | 35.391µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:13 | 200 | 32.697µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:15 | 200 | 33.196µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:20 | 200 | 33.643µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:23 | 200 | 31.888µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:25 | 200 | 35.021µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:30 | 200 | 31.245µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:33 | 200 | 32.829µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:35 | 200 | 31.554µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:40 | 200 | 35.112µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:43 | 200 | 51.767µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:45 | 200 | 34.203µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:50 | 200 | 31.99µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:53 | 200 | 31.413µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:00:55 | 200 | 33.944µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:00 | 200 | 35.866µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:03 | 200 | 43.585µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:05 | 200 | 48.568µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:10 | 200 | 35.003µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:13 | 200 | 32.747µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:15 | 200 | 32.836µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:20 | 200 | 31.946µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:23 | 200 | 31.214µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:25 | 200 | 32.481µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:30 | 200 | 34.73µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:33 | 200 | 33.139µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:35 | 200 | 43.545µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:40 | 200 | 33.759µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:43 | 200 | 33.468µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:45 | 200 | 33.07µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:50 | 200 | 30.637µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:53 | 200 | 33.525µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:01:55 | 200 | 32.101µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:00 | 200 | 36.872µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:03 | 200 | 34.455µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:05 | 200 | 40.857µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:10 | 200 | 32.452µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:13 | 200 | 36.336µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:15 | 200 | 32.277µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:20 | 200 | 38.688µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:23 | 200 | 31.363µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:25 | 200 | 31.782µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:30 | 200 | 37.062µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:33 | 200 | 32.523µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:35 | 200 | 33.348µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:40 | 200 | 34.534µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:43 | 200 | 32.235µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:45 | 200 | 34.135µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:50 | 200 | 33.007µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:53 | 200 | 36.103µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:02:55 | 200 | 32.416µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:00 | 200 | 40.025µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:03 | 200 | 33.449µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:05 | 200 | 32.015µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:10 | 200 | 32.57µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:13 | 200 | 31.701µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:15 | 200 | 32.111µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:20 | 200 | 32.144µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:23 | 200 | 33.353µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:25 | 200 | 32.075µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:30 | 200 | 35.661µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:33 | 200 | 30.738µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:35 | 200 | 32.68µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:40 | 200 | 38.362µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:43 | 200 | 33.164µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:45 | 200 | 32.668µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:50 | 200 | 33.061µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:53 | 200 | 33.789µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:03:55 | 200 | 32.114µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:04:00 | 200 | 33.497µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:04:03 | 200 | 32.742µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:04:05 | 200 | 34.455µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:04:10 | 200 | 34.305µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:04:13 | 200 | 32.814µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:04:15 | 200 | 34.744µs | 10.128.0.2 | GET "/health" [GIN] 2026/07/12 - 10:04:20 | 200 | 33.578µs | 10.128.0.2 | GET "/health" ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29728290-7sdjg.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29728305-2lkmp.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29728320-dzlkf.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29728335-zqlrm.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-controller-75bbfd978-nqzg4.log ==================== {"level":"info","ts":"2026-07-12T09:38:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:38:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:38:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:38:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:38:51Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:38:51Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:38:51Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"202e3864-0eb0-47fe-af90-6f4b119cd42a","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T09:38:51Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:38:51Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:38:51Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:38:52Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"202e3864-0eb0-47fe-af90-6f4b119cd42a","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:38:52Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"202e3864-0eb0-47fe-af90-6f4b119cd42a","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:38:52Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:38:52Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:39:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:39:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:39:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:39:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:40:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:40:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:40:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:40:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:41:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:41:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:41:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:41:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:42:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:42:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:42:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:42:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:43:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:43:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:43:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:43:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:43:52Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:43:52Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:43:52Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"9ebca4bd-9538-456d-ab82-48b3ab75a671","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T09:43:52Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:43:52Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:43:52Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:43:52Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"9ebca4bd-9538-456d-ab82-48b3ab75a671","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:43:52Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"9ebca4bd-9538-456d-ab82-48b3ab75a671","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:43:52Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:43:52Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:44:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:44:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:44:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:44:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:45:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:45:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:45:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:45:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:46:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:46:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:46:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:47:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:47:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:47:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:47:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:48:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:48:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:48:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:48:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:48:52Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:48:52Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:48:52Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"16f54a71-3bd8-4f1d-a220-26d71a994302","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T09:48:52Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:48:52Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:48:52Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:48:53Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"16f54a71-3bd8-4f1d-a220-26d71a994302","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:48:53Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"16f54a71-3bd8-4f1d-a220-26d71a994302","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:48:53Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:48:53Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:49:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:49:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:49:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:49:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:50:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:50:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:50:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:50:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:51:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:51:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:51:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:51:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:52:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:52:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:52:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:52:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:53:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:53:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:53:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:53:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:53:53Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:53:53Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:53:53Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"7996270b-8a75-4da9-b398-183a508b5a36","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T09:53:53Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:53:53Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:53:53Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:53:54Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"7996270b-8a75-4da9-b398-183a508b5a36","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:53:54Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"7996270b-8a75-4da9-b398-183a508b5a36","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:53:54Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:53:54Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:54:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:54:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:54:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:54:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:55:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:55:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:55:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:55:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:56:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:56:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:56:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:56:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:57:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:57:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:57:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:57:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:58:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:58:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:58:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:58:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:58:54Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:58:54Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:58:54Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"ba0b18a2-ecd5-4dc2-9141-d63f0febc14b","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T09:58:54Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:58:54Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:58:54Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:58:54Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"ba0b18a2-ecd5-4dc2-9141-d63f0febc14b","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:58:54Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"ba0b18a2-ecd5-4dc2-9141-d63f0febc14b","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T09:58:54Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T09:58:54Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T09:59:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:59:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T09:59:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T09:59:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T10:00:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T10:00:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T10:00:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T10:00:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T10:01:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T10:01:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T10:01:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T10:01:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T10:02:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T10:02:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T10:02:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T10:02:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T10:03:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T10:03:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T10:03:40Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ba61e35b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ba61e35b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ba61e35b"},"tenantIdentifier":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:40Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ba61e35b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ba61e35b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ba61e35b"},"tenantIdentifier":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:40Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"6c97f560-32b9-49e0-a951-29db4971e2e1","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","tenantID":"e2e-aigw-ba61e35b","subscriptionNamespace":"ai-tenant-e2e-aigw-ba61e35b","gatewayName":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:41Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ba61e35b","deploymentName":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:41Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ba61e35b","deploymentName":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:41Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ba61e35b","deploymentName":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"6c97f560-32b9-49e0-a951-29db4971e2e1","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"6c97f560-32b9-49e0-a951-29db4971e2e1","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:42Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ba61e35b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ba61e35b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ba61e35b"},"tenantIdentifier":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:42Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ba61e35b","deploymentName":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:42Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ba61e35b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ba61e35b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ba61e35b"},"tenantIdentifier":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:42Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ba61e35b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ba61e35b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ba61e35b"},"tenantIdentifier":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:42Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"f8ae56d0-4553-475e-aa40-e6119fd78113","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","tenantID":"e2e-aigw-ba61e35b","subscriptionNamespace":"ai-tenant-e2e-aigw-ba61e35b","gatewayName":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:42Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ba61e35b","deploymentName":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:42Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ba61e35b","deploymentName":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:42Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ba61e35b","deploymentName":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"f8ae56d0-4553-475e-aa40-e6119fd78113","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:43Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"f8ae56d0-4553-475e-aa40-e6119fd78113","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:43Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ba61e35b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ba61e35b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ba61e35b"},"tenantIdentifier":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:43Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ba61e35b","deploymentName":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T10:03:43Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} {"level":"info","ts":"2026-07-12T10:03:44Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ba61e35b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ba61e35b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ba61e35b"},"tenantIdentifier":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:44Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ba61e35b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ba61e35b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ba61e35b"},"tenantIdentifier":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Cleaning up MaaSAuthPolicy CRs","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"fb3d1609-2db6-4693-b88d-200a60f95227","namespace":"ai-tenant-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Cleaned up MaaSAuthPolicy CRs","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"fb3d1609-2db6-4693-b88d-200a60f95227","count":0} {"level":"info","ts":"2026-07-12T10:03:44Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ba61e35b/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ba61e35b","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ba61e35b","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ba61e35b"},"tenantIdentifier":"e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Cleaning up per-tenant maas-api resources","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"fb3d1609-2db6-4693-b88d-200a60f95227","tenant":"e2e-aigw-ba61e35b","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:44Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ba61e35b","deploymentName":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"fb3d1609-2db6-4693-b88d-200a60f95227","gvk":"Deployment","name":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"fb3d1609-2db6-4693-b88d-200a60f95227","gvk":"Service","name":"maas-api-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"fb3d1609-2db6-4693-b88d-200a60f95227","gvk":"HTTPRoute","name":"maas-api-e2e-aigw-ba61e35b-route"} {"level":"info","ts":"2026-07-12T10:03:44Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ba61e35b"},"namespace":"ai-tenant-e2e-aigw-ba61e35b","name":"default-tenant","reconcileID":"fb3d1609-2db6-4693-b88d-200a60f95227","gvk":"CronJob","name":"maas-api-key-cleanup-e2e-aigw-ba61e35b"} {"level":"info","ts":"2026-07-12T10:03:46Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ef12e752/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ef12e752","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ef12e752","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ef12e752"},"tenantIdentifier":"e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:46Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ef12e752/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ef12e752","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ef12e752","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ef12e752"},"tenantIdentifier":"e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:46Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"363e64af-5c43-42eb-bc62-6f2639f07d36","tenant":"ai-tenant-e2e-aigw-ef12e752/default-tenant","tenantID":"e2e-aigw-ef12e752","subscriptionNamespace":"ai-tenant-e2e-aigw-ef12e752","gatewayName":"e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ef12e752","deploymentName":"maas-api-e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ef12e752","deploymentName":"maas-api-e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:46Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ef12e752","deploymentName":"maas-api-e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"363e64af-5c43-42eb-bc62-6f2639f07d36","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"363e64af-5c43-42eb-bc62-6f2639f07d36","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:47Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ef12e752/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ef12e752","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ef12e752","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ef12e752"},"tenantIdentifier":"e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:47Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ef12e752","deploymentName":"maas-api-e2e-aigw-ef12e752"} {"level":"error","ts":"2026-07-12T10:03:47Z","msg":"Reconciler error","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"363e64af-5c43-42eb-bc62-6f2639f07d36","error":"Operation cannot be fulfilled on tenants.maas.opendatahub.io \"default-tenant\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"} {"level":"info","ts":"2026-07-12T10:03:47Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ef12e752/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ef12e752","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ef12e752","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ef12e752"},"tenantIdentifier":"e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:47Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ef12e752/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ef12e752","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ef12e752","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ef12e752"},"tenantIdentifier":"e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Cleaning up MaaSAuthPolicy CRs","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"089f945f-08d6-45c7-92ee-3ebe73731e77","namespace":"ai-tenant-e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Cleaned up MaaSAuthPolicy CRs","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"089f945f-08d6-45c7-92ee-3ebe73731e77","count":0} {"level":"info","ts":"2026-07-12T10:03:47Z","logger":"TenantIdentifierFor","msg":"Resolved tenant identifier from AITenant label","tenant":"ai-tenant-e2e-aigw-ef12e752/default-tenant","labels":{"ai-gateway.opendatahub.io/tenant":"e2e-aigw-ef12e752","app.kubernetes.io/managed-by":"maas-controller","app.kubernetes.io/part-of":"modelsasservice","maas.opendatahub.io/managed-by-aitenant":"true","maas.opendatahub.io/tenant-name":"e2e-aigw-ef12e752","maas.opendatahub.io/tenant-namespace":"ai-tenant-e2e-aigw-ef12e752"},"tenantIdentifier":"e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Cleaning up per-tenant maas-api resources","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"089f945f-08d6-45c7-92ee-3ebe73731e77","tenant":"e2e-aigw-ef12e752","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:47Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"e2e-aigw-ef12e752","deploymentName":"maas-api-e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"089f945f-08d6-45c7-92ee-3ebe73731e77","gvk":"Deployment","name":"maas-api-e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"089f945f-08d6-45c7-92ee-3ebe73731e77","gvk":"Service","name":"maas-api-e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"089f945f-08d6-45c7-92ee-3ebe73731e77","gvk":"HTTPRoute","name":"maas-api-e2e-aigw-ef12e752-route"} {"level":"info","ts":"2026-07-12T10:03:47Z","msg":"Deleted tenant resource","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"ai-tenant-e2e-aigw-ef12e752"},"namespace":"ai-tenant-e2e-aigw-ef12e752","name":"default-tenant","reconcileID":"089f945f-08d6-45c7-92ee-3ebe73731e77","gvk":"CronJob","name":"maas-api-key-cleanup-e2e-aigw-ef12e752"} {"level":"info","ts":"2026-07-12T10:03:54Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T10:03:54Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T10:03:54Z","msg":"Built platform params","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"17abf482-6d86-4bab-92e8-9b913588d9a3","tenant":"models-as-a-service/default-tenant","tenantID":"","subscriptionNamespace":"models-as-a-service","gatewayName":"maas-default-gateway"} {"level":"info","ts":"2026-07-12T10:03:54Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T10:03:54Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T10:03:54Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T10:03:56Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"17abf482-6d86-4bab-92e8-9b913588d9a3","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:56Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"17abf482-6d86-4bab-92e8-9b913588d9a3","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-12T10:03:56Z","logger":"TenantIdentifierFor","msg":"Using legacy/default tenant identifier (empty string)","tenant":"models-as-a-service/default-tenant","labels":null,"reason":"no AITenant label"} {"level":"info","ts":"2026-07-12T10:03:56Z","logger":"MaaSAPIDeploymentName","msg":"Generated deployment name","tenantID":"","deploymentName":"maas-api"} {"level":"info","ts":"2026-07-12T10:04:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"ai-tenants","purpose":"aitenant","phase":"Active"} {"level":"info","ts":"2026-07-12T10:04:13Z","logger":"setup","msg":"managed namespace already exists","namespace":"models-as-a-service","purpose":"subscription","phase":"Active"} ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-postgres-bb9d55498-x2zxw.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/mlflow-operator-controller-manager-76556d4b59-tmg29.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-ccb5b4bbc-w9kcp.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-serving-api-5679d6b4f-ww2g7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebook-controller-deployment-647b7d4d45-p44z4.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-model-controller-65fdb9c5c-rncwr.log ==================== 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"kuadrant-system"}, "namespace": "kuadrant-system", "name": "odh-trusted-ca-bundle", "reconcileID": "b0b30066-ac61-4813-adb1-e52a3adbfbf7", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "kuadrant-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kyverno"}, "namespace": "kyverno", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9ba241dd-a3f3-49c2-9a4d-6699fdc00d6b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kyverno"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-authentication-operator"}, "namespace": "openshift-authentication-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4b669289-43c1-4e07-bca3-1f5912ab8890", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-authentication-operator"} 2026-07-12T09:39:43Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "update": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-tam"}, "namespace": "openshift-backplane-tam", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1477ca10-1f2a-4dff-b73e-d6263a6e3107", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-tam"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-storage-operator"}, "namespace": "openshift-cluster-storage-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "01ccf43f-4fed-4d3c-9f22-121ef5e3a0af", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-storage-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-ogx-vector-stores-64f0"}, "namespace": "test-ogx-vector-stores-64f0", "name": "odh-trusted-ca-bundle", "reconcileID": "0a09eb0e-1e18-4cb1-8283-de7573521da2", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-ogx-vector-stores-64f0"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"models-as-a-service"}, "namespace": "models-as-a-service", "name": "openshift-service-ca.crt", "reconcileID": "7a8dc944-e4c9-4332-abb2-715141e0dd58", "ConfigMap": "openshift-service-ca.crt", "namespace": "models-as-a-service"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"oidc"}, "namespace": "oidc", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "644c91bb-251e-430b-8da4-22f8d40b3354", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "oidc"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console-user-settings"}, "namespace": "openshift-console-user-settings", "name": "openshift-service-ca.crt", "reconcileID": "deae481b-e062-47fe-93e0-abe04adf2720", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console-user-settings"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-insights"}, "namespace": "openshift-insights", "name": "openshift-service-ca.crt", "reconcileID": "200a084d-4c25-4abe-bc19-7768589ba99c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-insights"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane"}, "namespace": "openshift-backplane", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f600fec3-8fe5-461f-b692-1bea306a5a0e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-cee"}, "namespace": "openshift-backplane-cee", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f7bc8c9f-8829-420d-a071-6791194b6ba1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-cee"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"odh-ai-gateway-infra"}, "namespace": "odh-ai-gateway-infra", "name": "odh-trusted-ca-bundle", "reconcileID": "619da263-e9a9-4f3a-bb39-23be336b7529", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "odh-ai-gateway-infra"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "openshift-service-ca.crt", "reconcileID": "79679e43-b25f-492e-b968-12b8b07312f2", "ConfigMap": "openshift-service-ca.crt", "namespace": "rhoai-model-registries"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ebd65115-b8dd-4fb6-b70f-e0da9d972ab8", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "openshift-service-ca.crt", "reconcileID": "1b50b0d6-d58d-42a2-8415-5c5801d6f300", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-operator"}, "namespace": "openshift-network-operator", "name": "openshift-service-ca.crt", "reconcileID": "0aa39fbf-a1ec-4f62-bb32-3f3bb4a8b169", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-user-workload-monitoring"}, "namespace": "openshift-user-workload-monitoring", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3e3a225f-1a78-4374-8291-09e57c7b5d2f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-user-workload-monitoring"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-managed-scripts"}, "namespace": "openshift-backplane-managed-scripts", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6ab98111-5210-4751-84e9-69a0252c07a4", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-managed-scripts"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-node-tuning-operator"}, "namespace": "openshift-cluster-node-tuning-operator", "name": "openshift-service-ca.crt", "reconcileID": "b47b046f-7442-442e-86c6-8afac3542d8d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-node-tuning-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config-operator"}, "namespace": "openshift-config-operator", "name": "openshift-service-ca.crt", "reconcileID": "2efecba6-7c1d-4e2d-b907-6a7cc1628579", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-srep-ro"}, "namespace": "openshift-backplane-srep-ro", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f29e9bef-4d7e-4e05-bc8e-cdd1439157a4", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-srep-ro"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-machine-config-operator"}, "namespace": "openshift-machine-config-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "38f980eb-9039-4871-9bf6-1d5af3dbec9d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-machine-config-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-evalhub-mcp-deployment"}, "namespace": "test-evalhub-mcp-deployment", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b6c621bb-833d-4356-aa3c-ea7bd786e31f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-evalhub-mcp-deployment"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-infra"}, "namespace": "openshift-infra", "name": "openshift-service-ca.crt", "reconcileID": "e86d7a87-00e3-4619-939e-7d46f98a4835", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-infra"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-apiserver"}, "namespace": "openshift-kube-apiserver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b51017d6-1195-4986-8bea-dada161a1e71", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-apiserver"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-4f20af59"}, "namespace": "ai-tenant-e2e-aigw-4f20af59", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b5c20ad4-bf8a-4af2-9b9d-63f07885f16c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4f20af59"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-ai-agent"}, "namespace": "openshift-backplane-ai-agent", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "90a37fda-b0cc-4a2d-b3c1-2b36040d2040", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-ai-agent"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-config-managed"}, "namespace": "openshift-config-managed", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fa729c7f-6393-45d0-899d-27442b7b9b99", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-config-managed"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-af5eb82e"}, "namespace": "ai-tenant-e2e-aigw-af5eb82e", "name": "odh-trusted-ca-bundle", "reconcileID": "99406022-ebc9-4e2b-a7ec-e2f5e56791ee", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-af5eb82e"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"models-as-a-service"}, "namespace": "models-as-a-service", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2bd7379a-f386-4603-8a0e-e4e6cb2c0281", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "models-as-a-service"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-version"}, "namespace": "openshift-cluster-version", "name": "openshift-service-ca.crt", "reconcileID": "33838541-9e47-45a6-902e-7cafbea754dd", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-version"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "odh-trusted-ca-bundle", "reconcileID": "5a597aef-1d88-40ec-9316-1d0d3d5a0857", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-network-config-controller"}, "namespace": "openshift-cloud-network-config-controller", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b0378bc7-03d0-4513-90ba-db08790a320e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-network-config-controller"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-2kdmf"}, "namespace": "openshift-must-gather-2kdmf", "name": "openshift-service-ca.crt", "reconcileID": "fe121f3f-6e35-4464-a6b6-5e1fcab9bb29", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-2kdmf"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-package-operator"}, "namespace": "openshift-package-operator", "name": "openshift-service-ca.crt", "reconcileID": "1f70b9b2-acc0-4b7b-82e9-1c689e3273c9", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-package-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-4f20af59"}, "namespace": "ai-tenant-e2e-aigw-4f20af59", "name": "openshift-service-ca.crt", "reconcileID": "9fd47346-c508-466a-bc4b-1ef61c1de6bb", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-4f20af59"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"open-cluster-management-2r2qdb90ltvhjc43lp6iqc6aqb7bgjda"}, "namespace": "open-cluster-management-2r2qdb90ltvhjc43lp6iqc6aqb7bgjda", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "190f46e3-fe4b-498a-9976-9c77ecd5bbd0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "open-cluster-management-2r2qdb90ltvhjc43lp6iqc6aqb7bgjda"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-4a1dd36b"}, "namespace": "ai-tenant-e2e-aigw-4a1dd36b", "name": "openshift-service-ca.crt", "reconcileID": "72422b3d-2c78-4d7a-a503-4b24d2d2de0b", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-4a1dd36b"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kuadrant-system"}, "namespace": "kuadrant-system", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "980a6cde-4be5-45b8-a03d-f0f101344a76", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kuadrant-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-image-registry"}, "namespace": "openshift-image-registry", "name": "openshift-service-ca.crt", "reconcileID": "f14a8fbd-5a55-49de-b444-81c8f98ef633", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-image-registry"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-node"}, "namespace": "openshift-node", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "8e411b36-b0d8-4677-aeed-0c6b9528a1d6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-node"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"wva-system"}, "namespace": "wva-system", "name": "odh-trusted-ca-bundle", "reconcileID": "96731a75-ab99-4108-8460-637c61deeadc", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "wva-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-credential-operator"}, "namespace": "openshift-cloud-credential-operator", "name": "openshift-service-ca.crt", "reconcileID": "a2386641-cbb2-4bc9-a57c-fd4b9815e0fc", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-credential-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-insights"}, "namespace": "openshift-insights", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dc5e2f12-bc10-49bc-9bed-e45e3a73e91f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-insights"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-scheduler"}, "namespace": "openshift-kube-scheduler", "name": "openshift-service-ca.crt", "reconcileID": "cc6e4a37-9699-45a6-bc69-ce672fb8d2ad", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-scheduler"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-t4hxf"}, "namespace": "openshift-must-gather-t4hxf", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "92a29620-32a7-4172-92fb-e14eecd64f3d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-t4hxf"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-6add620d"}, "namespace": "ai-tenant-e2e-aigw-6add620d", "name": "odh-trusted-ca-bundle", "reconcileID": "0d398c5c-4ef0-46f8-bf5e-6439fdb05be2", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-6add620d"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-t4hxf"}, "namespace": "openshift-must-gather-t4hxf", "name": "openshift-service-ca.crt", "reconcileID": "55125fb2-7f44-489e-b85c-780c615265e4", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-t4hxf"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"odh-ai-gateway-infra"}, "namespace": "odh-ai-gateway-infra", "name": "openshift-service-ca.crt", "reconcileID": "c8076ac4-080b-4a93-bf6d-371557d57d21", "ConfigMap": "openshift-service-ca.crt", "namespace": "odh-ai-gateway-infra"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-controller-manager-operator"}, "namespace": "openshift-controller-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "0bc31f64-b2ee-43e0-b08e-9d442011f44b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-controller-manager-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-etcd"}, "namespace": "openshift-etcd", "name": "openshift-service-ca.crt", "reconcileID": "427d65ef-15a0-4f5c-9aff-7e8dfa2260b6", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-etcd"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-route-controller-manager"}, "namespace": "openshift-route-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "fc578d49-9060-456c-be30-2dc30f41fb44", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-route-controller-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fafa0bdc-292b-4f62-b69e-d38ffa761037", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "rhoai-model-registries"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-a124ef78"}, "namespace": "ai-tenant-e2e-aigw-a124ef78", "name": "openshift-service-ca.crt", "reconcileID": "974003b4-e524-463c-b65e-cf735bb4cde4", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-a124ef78"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-dns"}, "namespace": "openshift-dns", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cb00cc2a-e3b2-41ab-9efa-6d9d6b5c654d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-dns"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift"}, "namespace": "openshift", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0f94ab2d-0a12-4634-8714-4a5ef9687fae", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-e600492d"}, "namespace": "ai-tenant-e2e-aigw-e600492d", "name": "odh-trusted-ca-bundle", "reconcileID": "0bb0c660-3072-4bf2-a9f1-4bf8413397a3", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-e600492d"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-config"}, "namespace": "openshift-config", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2441fe2a-98a2-4d95-afe7-18f604a66f01", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-config"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-gzm9m"}, "namespace": "openshift-must-gather-gzm9m", "name": "openshift-service-ca.crt", "reconcileID": "b583fdab-18ff-4bc5-8c9a-2d294eddaac0", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-gzm9m"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-97xps"}, "namespace": "openshift-must-gather-97xps", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "09d4f220-63f7-4c32-9720-70f310f60297", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-97xps"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-diagnostics"}, "namespace": "openshift-network-diagnostics", "name": "openshift-service-ca.crt", "reconcileID": "b34266f4-8512-4ca3-8963-18b4e973393f", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-diagnostics"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-operator-lifecycle-manager"}, "namespace": "openshift-operator-lifecycle-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1e66eac5-1610-40f2-a896-91e4e49d289b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-operator-lifecycle-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operators-redhat"}, "namespace": "openshift-operators-redhat", "name": "openshift-service-ca.crt", "reconcileID": "b1f9fb47-7c6d-409e-9200-ff053154b7d8", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operators-redhat"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-8ed53301"}, "namespace": "ai-tenant-e2e-aigw-8ed53301", "name": "odh-trusted-ca-bundle", "reconcileID": "b702ec85-a6ae-4252-995f-9986ed493b62", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-8ed53301"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-authentication-operator"}, "namespace": "openshift-authentication-operator", "name": "openshift-service-ca.crt", "reconcileID": "d9322f06-727d-476c-b1c6-6c1890fe26ea", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-authentication-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config-managed"}, "namespace": "openshift-config-managed", "name": "openshift-service-ca.crt", "reconcileID": "3b89a389-6b7c-4002-8dca-6d5cbd8e0661", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config-managed"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-config-operator"}, "namespace": "openshift-config-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6d0185f3-83ba-45b0-9ebf-efd0069cab00", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-config-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-image-registry"}, "namespace": "openshift-image-registry", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ae051217-9503-48b0-952b-a3831e97478b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-image-registry"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ingress-canary"}, "namespace": "openshift-ingress-canary", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "73353eb5-be37-4ad0-9d59-c52dc475644b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ingress-canary"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-scheduler-operator"}, "namespace": "openshift-kube-scheduler-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "404aab1c-5ced-4988-92d1-dbb1bd168705", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-scheduler-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift"}, "namespace": "openshift", "name": "openshift-service-ca.crt", "reconcileID": "97389398-29ce-46b8-9432-c53d288f6875", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-069d963f"}, "namespace": "ai-tenant-e2e-aigw-069d963f", "name": "openshift-service-ca.crt", "reconcileID": "8ab5cee8-470e-4f7d-b515-2fe05b08b63a", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-069d963f"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"cert-manager-operator"}, "namespace": "cert-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "8ff53a9c-d299-46b9-9a96-9ad91df28b21", "ConfigMap": "openshift-service-ca.crt", "namespace": "cert-manager-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-authentication"}, "namespace": "openshift-authentication", "name": "openshift-service-ca.crt", "reconcileID": "8f086dd2-1364-4efd-af41-2c8293d55878", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-authentication"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-csm"}, "namespace": "openshift-backplane-csm", "name": "openshift-service-ca.crt", "reconcileID": "5bcdd905-514b-4017-afff-8f36358f10e7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-csm"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-operator"}, "namespace": "openshift-must-gather-operator", "name": "openshift-service-ca.crt", "reconcileID": "eadc2374-60e9-4f4e-9c6f-efd79077ea54", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-88ae976f"}, "namespace": "ai-tenant-e2e-aigw-88ae976f", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e391d378-0001-4c24-9723-e282e2bb1657", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-88ae976f"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"cert-manager"}, "namespace": "cert-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "92a5881b-5d48-4820-a967-e289c9e8366a", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "cert-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-apiserver-operator"}, "namespace": "openshift-apiserver-operator", "name": "openshift-service-ca.crt", "reconcileID": "d491de8b-1581-4a42-b1cb-d4507431d5c6", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-apiserver-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-controller-manager"}, "namespace": "openshift-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e90b954c-815d-4203-b391-79de62e8674d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-controller-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-4pg7b"}, "namespace": "openshift-must-gather-4pg7b", "name": "openshift-service-ca.crt", "reconcileID": "cc59c726-210e-4205-9901-2dacd4258a95", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-4pg7b"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"redhat-ods-operator"}, "namespace": "redhat-ods-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "92376032-a90c-414d-b1d4-f74a29e05771", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "redhat-ods-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"cert-manager"}, "namespace": "cert-manager", "name": "odh-trusted-ca-bundle", "reconcileID": "7342e48f-3cc3-4bdf-b312-befb37217e8e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "cert-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-apiserver"}, "namespace": "openshift-apiserver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b4d57522-f996-40cc-bc58-bc43e1117081", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-apiserver"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-pxpxm"}, "namespace": "openshift-must-gather-pxpxm", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "496089b3-f32d-41be-970f-1226445defc3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-pxpxm"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-069d963f"}, "namespace": "ai-tenant-e2e-aigw-069d963f", "name": "odh-trusted-ca-bundle", "reconcileID": "4eb06802-7829-44c0-b207-9ca402810604", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-069d963f"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openldap"}, "namespace": "openldap", "name": "openshift-service-ca.crt", "reconcileID": "21bd36db-886d-49a2-ad0d-4c71b5193f3c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openldap"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-machine-api"}, "namespace": "openshift-machine-api", "name": "openshift-service-ca.crt", "reconcileID": "2f26803d-42ab-44fb-b4f2-09da73bf1143", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-machine-api"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-tam"}, "namespace": "openshift-backplane-tam", "name": "openshift-service-ca.crt", "reconcileID": "051588ba-4292-4bb7-9021-b977bac8c328", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-tam"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-monitoring"}, "namespace": "redhat-ods-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "d4d51393-4350-417f-9d86-57c6d6bdf3c9", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-monitoring"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-e072fffe"}, "namespace": "ai-tenant-e2e-aigw-e072fffe", "name": "odh-trusted-ca-bundle", "reconcileID": "019baafb-14f9-4128-b2ec-5cd1355f625a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-e072fffe"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-network-config-controller"}, "namespace": "openshift-cloud-network-config-controller", "name": "openshift-service-ca.crt", "reconcileID": "5fe0af34-3929-488b-be03-d15ceaca36c2", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-network-config-controller"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-44lbb"}, "namespace": "openshift-must-gather-44lbb", "name": "openshift-service-ca.crt", "reconcileID": "ff987ae3-46eb-4a5d-a97d-18bcb870ebbc", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-44lbb"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-4f20af59"}, "namespace": "ai-tenant-e2e-aigw-4f20af59", "name": "odh-trusted-ca-bundle", "reconcileID": "a7dc773f-6f2c-4c4f-8dc7-705c1433148c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-4f20af59"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-controller-manager"}, "namespace": "openshift-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "9df06db7-498a-44e4-9eb1-93de0fd2b24f", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-controller-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-operators-redhat"}, "namespace": "openshift-operators-redhat", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "60621924-b9d3-4ce6-a64f-cf2a684db6f9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-operators-redhat"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "openshift-service-ca.crt", "reconcileID": "751ad8c3-5453-4834-b8d4-c681225d17a4", "ConfigMap": "openshift-service-ca.crt", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-mobb"}, "namespace": "openshift-backplane-mobb", "name": "openshift-service-ca.crt", "reconcileID": "0279357e-ea1b-4484-8290-30a610b1ee47", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-mobb"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-csi-drivers"}, "namespace": "openshift-cluster-csi-drivers", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e62bbe80-5611-4b68-bf7c-5ccbcdd2f4e7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-csi-drivers"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-4pg7b"}, "namespace": "openshift-must-gather-4pg7b", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "54d4ffde-e45a-44f5-9962-6044932f43ca", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-4pg7b"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-deployment-validation-operator"}, "namespace": "openshift-deployment-validation-operator", "name": "openshift-service-ca.crt", "reconcileID": "e63880c6-d98f-4e38-9bbf-dc62f94ee431", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-deployment-validation-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-8ed53301"}, "namespace": "ai-tenant-e2e-aigw-8ed53301", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "33bd380b-ad87-4681-92d5-c32f5dfe09dc", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-8ed53301"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-machine-approver"}, "namespace": "openshift-cluster-machine-approver", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f5b9b56a-89de-4a9e-acc1-0d9eea57ec79", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-machine-approver"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-gzm9m"}, "namespace": "openshift-must-gather-gzm9m", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9eef8d93-238b-4693-8eba-14de88619778", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-gzm9m"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-package-operator"}, "namespace": "openshift-package-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c4d45f86-c244-4a91-b9b9-c0c2213db099", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-package-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-apiserver"}, "namespace": "openshift-apiserver", "name": "openshift-service-ca.crt", "reconcileID": "aaf6b397-6e4a-4dbb-b09a-9ada4d7f3dfe", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-apiserver"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kueue-managed-d989jnk35hkc74rp4pag"}, "namespace": "test-kueue-managed-d989jnk35hkc74rp4pag", "name": "odh-trusted-ca-bundle", "reconcileID": "2dbe5730-1682-4d81-97af-9d942c555cfc", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kueue-managed-d989jnk35hkc74rp4pag"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-ogx-vector-stores-64f0"}, "namespace": "test-ogx-vector-stores-64f0", "name": "openshift-service-ca.crt", "reconcileID": "9c1309fe-d46c-49c0-b942-941e13a98abe", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-ogx-vector-stores-64f0"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "odh-trusted-ca-bundle", "reconcileID": "2db9c122-ddb8-46bd-b237-39107ba60f41", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"open-cluster-management-agent-addon"}, "namespace": "open-cluster-management-agent-addon", "name": "odh-trusted-ca-bundle", "reconcileID": "31ac2bdb-e378-4f18-bacd-8bf167133051", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "open-cluster-management-agent-addon"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-277ae597"}, "namespace": "ai-tenant-e2e-aigw-277ae597", "name": "odh-trusted-ca-bundle", "reconcileID": "db07d7e2-4562-47c1-afda-f233a93a2625", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-277ae597"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-af5eb82e"}, "namespace": "ai-tenant-e2e-aigw-af5eb82e", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "173e255e-1b22-4a4f-a611-493720533119", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-af5eb82e"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-srep"}, "namespace": "openshift-backplane-srep", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d994a685-8d6e-4bf2-9f11-01287f5e5039", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-srep"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-service-ca"}, "namespace": "openshift-service-ca", "name": "openshift-service-ca.crt", "reconcileID": "e5ff9f35-c62b-4789-912a-eef9dbffa939", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-service-ca"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-controller-manager-operator"}, "namespace": "openshift-kube-controller-manager-operator", "name": "openshift-service-ca.crt", "reconcileID": "630f4120-50e7-4cc0-9580-02cac9412b37", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-controller-manager-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"workload-variant-autoscaler-system"}, "namespace": "workload-variant-autoscaler-system", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6f9ce69b-ca5d-461f-a1ef-93b10ebd5212", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "workload-variant-autoscaler-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"oidc"}, "namespace": "oidc", "name": "odh-trusted-ca-bundle", "reconcileID": "be4edcf2-5c10-4423-b8b9-105c5b115926", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "oidc"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-node-identity"}, "namespace": "openshift-network-node-identity", "name": "openshift-service-ca.crt", "reconcileID": "46af9f0b-6544-41d8-a3e2-2590c140f691", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-node-identity"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "odh-trusted-ca-bundle", "reconcileID": "e3e6e887-c809-4d12-855f-4e3ae939bb32", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "rhods-notebooks"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"cert-manager"}, "namespace": "cert-manager", "name": "openshift-service-ca.crt", "reconcileID": "352d526d-3145-49e9-a95d-385740f031f7", "ConfigMap": "openshift-service-ca.crt", "namespace": "cert-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-csi-drivers"}, "namespace": "openshift-cluster-csi-drivers", "name": "openshift-service-ca.crt", "reconcileID": "2e2de4b9-6d4e-4e83-b234-2dbe8d2deb10", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-csi-drivers"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-apiserver"}, "namespace": "openshift-kube-apiserver", "name": "openshift-service-ca.crt", "reconcileID": "24f0e0ba-d403-4365-992b-0b59bf361913", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-apiserver"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-88ae976f"}, "namespace": "ai-tenant-e2e-aigw-88ae976f", "name": "odh-trusted-ca-bundle", "reconcileID": "e20b7b0d-550c-4da7-ba83-bcddfe0a598f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-88ae976f"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenants"}, "namespace": "ai-tenants", "name": "openshift-service-ca.crt", "reconcileID": "2fee5be9-8cef-457a-a42b-11f1a17ca90e", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenants"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cloud-controller-manager"}, "namespace": "openshift-cloud-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "648cad7a-7e26-4da6-a3a8-9fc37298de35", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cloud-controller-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"wva-system"}, "namespace": "wva-system", "name": "openshift-service-ca.crt", "reconcileID": "4c36eddc-dbcd-405c-bc93-707379e7c274", "ConfigMap": "openshift-service-ca.crt", "namespace": "wva-system"} 2026-07-12T09:39:43Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "ConfigMap:": "opt-125m-standard-cpu-metrics-dashboard"} 2026-07-12T09:39:43Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:39:43Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-controller-manager"}, "namespace": "openshift-kube-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6eceb7ef-e9ab-4cb3-8d1c-733082706ef3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-controller-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kube-public"}, "namespace": "kube-public", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5d4adc51-3cc0-4473-a8a3-91ced69e542f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kube-public"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-controller-manager"}, "namespace": "openshift-cloud-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d37a1ac2-330e-4f13-af4d-1274844dd112", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-controller-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-node-tuning-operator"}, "namespace": "openshift-cluster-node-tuning-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "acf90353-1db7-4d8f-9904-122072cdab0c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-node-tuning-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "de6e25e4-fedd-4807-b675-506df64bfe7d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ingress-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-scheduler"}, "namespace": "openshift-kube-scheduler", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "6d2bc8d0-2c6a-44d7-8375-c167a34faf43", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-scheduler"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-storage-version-migrator"}, "namespace": "openshift-kube-storage-version-migrator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "40d394ad-9ab5-4a80-928b-44b3680a9c6b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-storage-version-migrator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-node-lease"}, "namespace": "kube-node-lease", "name": "openshift-service-ca.crt", "reconcileID": "dfe0e7f9-ffa1-4fb2-b35c-a72de9a229cc", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-node-lease"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b7ae66ce-c6a4-43b5-aebf-d1355ec9599b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ingress"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console-operator"}, "namespace": "openshift-console-operator", "name": "openshift-service-ca.crt", "reconcileID": "2b5f1417-d662-4a98-9302-7762c19fb0a1", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "openshift-service-ca.crt", "reconcileID": "d4580875-32cc-479c-8067-5210991ed672", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-ovn-kubernetes"}, "namespace": "openshift-ovn-kubernetes", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d0b8b960-400a-4ce0-958d-52146b4a24ce", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-ovn-kubernetes"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"wva-system"}, "namespace": "wva-system", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "48e6d921-2782-4fa3-93e8-e843abdbcec3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "wva-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-storage-version-migrator"}, "namespace": "openshift-kube-storage-version-migrator", "name": "openshift-service-ca.crt", "reconcileID": "508d1813-44af-4bc0-8e90-b1c4fcd66061", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-storage-version-migrator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-operator"}, "namespace": "openshift-must-gather-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9ab1c1a9-c5f6-430d-905b-09360cc16634", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "89dcd5f7-fd13-4a55-9f02-c6423135b313", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "rhods-notebooks"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-6add620d"}, "namespace": "ai-tenant-e2e-aigw-6add620d", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4bebfe15-4a7b-400a-909f-bdd947a1f125", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-6add620d"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-monitoring"}, "namespace": "openshift-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "6dd82942-cc55-40ba-8518-dca0a87fe1e7", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-monitoring"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kuadrant-system"}, "namespace": "kuadrant-system", "name": "openshift-service-ca.crt", "reconcileID": "a8ad31af-15ba-46bf-8d37-aff0557b1c71", "ConfigMap": "openshift-service-ca.crt", "namespace": "kuadrant-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-storage-operator"}, "namespace": "openshift-cluster-storage-operator", "name": "openshift-service-ca.crt", "reconcileID": "bbf03f7b-2875-47cb-90c8-20c2b8461e1a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-storage-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-2kdmf"}, "namespace": "openshift-must-gather-2kdmf", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "53304423-dd72-4b8f-8e95-613602d11818", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-2kdmf"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operators"}, "namespace": "openshift-operators", "name": "openshift-service-ca.crt", "reconcileID": "46cffbab-fb31-4061-9181-6a899a2e2c9c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operators"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-6add620d"}, "namespace": "ai-tenant-e2e-aigw-6add620d", "name": "openshift-service-ca.crt", "reconcileID": "d7c8dc2b-b145-4d3d-a155-d217a7d247a4", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-6add620d"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-node-identity"}, "namespace": "openshift-network-node-identity", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fc1db550-b102-45a0-abb6-0115a146dbe3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-node-identity"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-customer-monitoring"}, "namespace": "openshift-customer-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "8b2a0acd-c73f-4d0b-9afb-d772b4b8e132", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-customer-monitoring"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-scheduler-operator"}, "namespace": "openshift-kube-scheduler-operator", "name": "openshift-service-ca.crt", "reconcileID": "513f6dd8-9561-48d9-adad-7d7b2b21c675", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-scheduler-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-marketplace"}, "namespace": "openshift-marketplace", "name": "openshift-service-ca.crt", "reconcileID": "4bd557f0-42cb-4129-a78a-33515a7d039e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-marketplace"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-4tx7s"}, "namespace": "openshift-must-gather-4tx7s", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fecde64c-14d5-4649-9f99-d0028964f3dd", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-4tx7s"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"workload-variant-autoscaler-system"}, "namespace": "workload-variant-autoscaler-system", "name": "openshift-service-ca.crt", "reconcileID": "895090e6-781f-4512-a097-c38219a269c6", "ConfigMap": "openshift-service-ca.crt", "namespace": "workload-variant-autoscaler-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-mcs-tier-two"}, "namespace": "openshift-backplane-mcs-tier-two", "name": "openshift-service-ca.crt", "reconcileID": "9f8d8236-d920-40e8-9dbc-7a4eb102b8b1", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-mcs-tier-two"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-machine-config-operator"}, "namespace": "openshift-machine-config-operator", "name": "openshift-service-ca.crt", "reconcileID": "ac98c6d1-d243-4c76-88a6-925c74210b03", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-machine-config-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-must-gather-44lbb"}, "namespace": "openshift-must-gather-44lbb", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7bcbe316-ce3f-49c2-87eb-838b65f0604c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-must-gather-44lbb"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-operator"}, "namespace": "openshift-network-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "75839a93-71b6-4751-a0f1-09534e276867", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-user-workload-monitoring"}, "namespace": "openshift-user-workload-monitoring", "name": "openshift-service-ca.crt", "reconcileID": "c31be733-4339-4585-90c7-84fabf7ea6d6", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-user-workload-monitoring"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-network-console"}, "namespace": "openshift-network-console", "name": "openshift-service-ca.crt", "reconcileID": "3042f498-53bf-4a87-9e1b-832957d000ae", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-network-console"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-service-ca-operator"}, "namespace": "openshift-service-ca-operator", "name": "openshift-service-ca.crt", "reconcileID": "e96ddf4b-2e65-43fc-a80f-59955114ed90", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-service-ca-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-277ae597"}, "namespace": "ai-tenant-e2e-aigw-277ae597", "name": "openshift-service-ca.crt", "reconcileID": "c7a87a3a-c08d-47b7-9c54-e85bbdbbf4ba", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-277ae597"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-a124ef78"}, "namespace": "ai-tenant-e2e-aigw-a124ef78", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d0669d5b-6484-4e62-8ebf-3c9322483855", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-a124ef78"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-multus"}, "namespace": "openshift-multus", "name": "openshift-service-ca.crt", "reconcileID": "2dc820e7-4732-49a6-9d7f-6005ea65d572", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-multus"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "openshift-service-ca.crt", "reconcileID": "2179e81f-78ee-4f59-b3c4-4ea51ed9c069", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-applications"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-069d963f"}, "namespace": "ai-tenant-e2e-aigw-069d963f", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d0c17697-18b9-4ede-b859-e202bf9c63b6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-069d963f"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-cse"}, "namespace": "openshift-backplane-cse", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c6eec618-489d-470e-8676-5da3acf7d5e0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-cse"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-evalhub-tenant-a-testevalhubmcpdeployment-1b7577"}, "namespace": "test-evalhub-tenant-a-testevalhubmcpdeployment-1b7577", "name": "openshift-service-ca.crt", "reconcileID": "e0c65104-aed7-431b-9560-75910e4dfc47", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-evalhub-tenant-a-testevalhubmcpdeployment-1b7577"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-route-controller-manager"}, "namespace": "openshift-route-controller-manager", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bcca6a14-061a-4e63-8ec1-a273f1d578a5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-route-controller-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"default"}, "namespace": "default", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c79f47ab-a5e1-413f-b498-ea0d87df1560", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "default"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-ai-agent"}, "namespace": "openshift-backplane-ai-agent", "name": "openshift-service-ca.crt", "reconcileID": "5337a9b1-00b1-46a5-8dc0-a2c991bb4797", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-ai-agent"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-lpsre"}, "namespace": "openshift-backplane-lpsre", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e034c0b0-07fc-4794-abde-b0320b2d3ec1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-lpsre"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-diagnostics"}, "namespace": "openshift-network-diagnostics", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "bfe0adbd-5e5b-4fd0-a6f5-bf5a0a514a6b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-diagnostics"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "odh-trusted-ca-bundle", "reconcileID": "706f14b9-ad61-4bef-9e3c-46ac38ac5c0b", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-host-network"}, "namespace": "openshift-host-network", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "de467be3-d6f6-451e-a684-a265e825ade9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-host-network"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-managed-scripts"}, "namespace": "openshift-backplane-managed-scripts", "name": "openshift-service-ca.crt", "reconcileID": "82671529-e37b-4d9d-9ba8-de9a102e65f0", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-managed-scripts"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openldap"}, "namespace": "openldap", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "571143e0-5422-41af-ac7b-6e94629e40c3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openldap"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-console-user-settings"}, "namespace": "openshift-console-user-settings", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0067edbb-eb3f-4ae1-b375-400e205a471d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-console-user-settings"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-jobset-operator"}, "namespace": "openshift-jobset-operator", "name": "openshift-service-ca.crt", "reconcileID": "195227aa-2e14-43c2-b06c-6b246e345e0e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-jobset-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"rhoai-model-registries"}, "namespace": "rhoai-model-registries", "name": "odh-trusted-ca-bundle", "reconcileID": "690492af-3751-4b2f-8116-e80ea7b55659", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "rhoai-model-registries"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-authentication"}, "namespace": "openshift-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "cdc44f4e-a9b0-4171-8fda-74f846e8ff09", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-authentication"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"redhat-ods-operator"}, "namespace": "redhat-ods-operator", "name": "odh-trusted-ca-bundle", "reconcileID": "4c3b87d2-8d9d-46bb-b171-964cb16085d9", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "redhat-ods-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-mcs-tier-two"}, "namespace": "openshift-backplane-mcs-tier-two", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c0cae431-6f4f-4f0c-9c83-d1387a993c6b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-mcs-tier-two"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"models-as-a-service"}, "namespace": "models-as-a-service", "name": "odh-trusted-ca-bundle", "reconcileID": "dc1db1a8-8a8b-4f8e-ae2b-691075db05e5", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "models-as-a-service"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"redhat-ods-monitoring"}, "namespace": "redhat-ods-monitoring", "name": "odh-trusted-ca-bundle", "reconcileID": "9789fdc2-687d-4fc9-b6fa-d0e036fa7d50", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "redhat-ods-monitoring"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-operator"}, "namespace": "redhat-ods-operator", "name": "openshift-service-ca.crt", "reconcileID": "6f36308a-8739-4edd-9794-6fe1b84c5440", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"odh-ai-gateway-infra"}, "namespace": "odh-ai-gateway-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ebd5dd2b-b293-4ef9-814f-9205a5c425d6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "odh-ai-gateway-infra"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"open-cluster-management-agent-addon"}, "namespace": "open-cluster-management-agent-addon", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "16535ee3-6848-43fb-b8af-c2900febde62", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "open-cluster-management-agent-addon"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-apiserver-operator"}, "namespace": "openshift-kube-apiserver-operator", "name": "openshift-service-ca.crt", "reconcileID": "9d2432bc-d3da-4be5-b506-2b09433a7e09", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-apiserver-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenants"}, "namespace": "ai-tenants", "name": "odh-trusted-ca-bundle", "reconcileID": "9ef58b57-07ff-4ca5-b949-cc9f399cf47f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenants"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-csm"}, "namespace": "openshift-backplane-csm", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3a54b386-a202-4166-8573-e2fcb1f9bd8d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-csm"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-lpsre"}, "namespace": "openshift-backplane-lpsre", "name": "openshift-service-ca.crt", "reconcileID": "a2775982-1bbb-4153-aa2b-113f50a4d684", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-lpsre"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress-canary"}, "namespace": "openshift-ingress-canary", "name": "openshift-service-ca.crt", "reconcileID": "856f3516-e6e1-4b10-ad8c-39997d6dc1de", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress-canary"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-cf933f2a"}, "namespace": "ai-tenant-e2e-aigw-cf933f2a", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b3cb3b1f-428a-4de5-9305-cc8a80c08851", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-cf933f2a"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-keda"}, "namespace": "openshift-keda", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "559a136d-57c4-4ea0-a5af-6c86131ceb6e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-keda"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "83c1a2c7-d862-42c0-9f33-f298903741e6", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "redhat-ods-applications"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"redhat-ods-monitoring"}, "namespace": "redhat-ods-monitoring", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "87338924-b2e6-4206-89c6-e20ecb1c72c3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "redhat-ods-monitoring"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"cert-manager-operator"}, "namespace": "cert-manager-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7cc2ce8c-170a-4c00-a17d-091012b235c7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "cert-manager-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kube-system"}, "namespace": "kube-system", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4af7f5c1-d7cb-47e6-860c-8c48b6579281", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kube-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-logging"}, "namespace": "openshift-logging", "name": "openshift-service-ca.crt", "reconcileID": "cd699530-26f2-43b6-bd77-768cc1fc4666", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-logging"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-controller-manager"}, "namespace": "openshift-kube-controller-manager", "name": "openshift-service-ca.crt", "reconcileID": "57759f82-7758-49b8-a38e-6529abe5c115", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-controller-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-pxpxm"}, "namespace": "openshift-must-gather-pxpxm", "name": "openshift-service-ca.crt", "reconcileID": "e9da93d5-fbef-4ef4-80ad-f7be766f697a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-pxpxm"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"open-cluster-management-agent-addon"}, "namespace": "open-cluster-management-agent-addon", "name": "openshift-service-ca.crt", "reconcileID": "8c51cd9b-c013-4ab7-9eb9-8500a6e67164", "ConfigMap": "openshift-service-ca.crt", "namespace": "open-cluster-management-agent-addon"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kueue-managed-d989jnk35hkc74rp4pag"}, "namespace": "test-kueue-managed-d989jnk35hkc74rp4pag", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a89c1838-553a-430e-8020-10230f625c9b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kueue-managed-d989jnk35hkc74rp4pag"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-operator-lifecycle-manager"}, "namespace": "openshift-operator-lifecycle-manager", "name": "openshift-service-ca.crt", "reconcileID": "e32c3113-c3bf-4107-89ce-3808cebe36cc", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-operator-lifecycle-manager"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"openldap"}, "namespace": "openldap", "name": "odh-trusted-ca-bundle", "reconcileID": "e186f467-af5a-4bb5-bb8a-fdca36633780", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "openldap"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "openshift-service-ca.crt", "reconcileID": "9e6cb488-0514-4f0f-97e6-7268e99eff94", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ingress-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-cee"}, "namespace": "openshift-backplane-cee", "name": "openshift-service-ca.crt", "reconcileID": "20a1819f-2887-4ae8-8ca5-bb6dee973a30", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-cee"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-samples-operator"}, "namespace": "openshift-cluster-samples-operator", "name": "openshift-service-ca.crt", "reconcileID": "17e46232-c412-4f60-a193-27c2d9bc847c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-samples-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"dedicated-admin"}, "namespace": "dedicated-admin", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c25e668b-2afb-4792-a11c-a3fb2a41f7db", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "dedicated-admin"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-cluster-machine-approver"}, "namespace": "openshift-cluster-machine-approver", "name": "openshift-service-ca.crt", "reconcileID": "905a16d1-537b-4fb4-9d90-8025ff4ee53d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-cluster-machine-approver"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-controller-manager-operator"}, "namespace": "openshift-controller-manager-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e74da3a1-46e2-44f9-909e-620201b6a22d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-controller-manager-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-lws-operator"}, "namespace": "openshift-lws-operator", "name": "openshift-service-ca.crt", "reconcileID": "260493a8-a61e-417c-91b6-6228e27a36c2", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-lws-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-marketplace"}, "namespace": "openshift-marketplace", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a426738a-7b30-4430-9207-577a90c1ae01", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-marketplace"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"default"}, "namespace": "default", "name": "openshift-service-ca.crt", "reconcileID": "202a40df-c2d5-4e25-826b-ffda6dc24dd0", "ConfigMap": "openshift-service-ca.crt", "namespace": "default"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-logging"}, "namespace": "openshift-logging", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "47ff5443-ad3b-412c-ad7a-386d28db2e26", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-logging"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-monitoring"}, "namespace": "openshift-monitoring", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a867f969-053a-477c-8753-34b7bd186a03", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-monitoring"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-evalhub-mcp-deployment"}, "namespace": "test-evalhub-mcp-deployment", "name": "odh-trusted-ca-bundle", "reconcileID": "37c1ef72-dc37-4129-ac79-fce2e528cb8c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-evalhub-mcp-deployment"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-evalhub-tenant-a-testevalhubmcpdeployment-1b7577"}, "namespace": "test-evalhub-tenant-a-testevalhubmcpdeployment-1b7577", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "08c0d7ce-6b12-4d98-a2b8-14e3fcb55736", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-evalhub-tenant-a-testevalhubmcpdeployment-1b7577"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"kube-node-lease"}, "namespace": "kube-node-lease", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5bd0d43e-0007-49a0-92ff-bd0adc9a4570", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "kube-node-lease"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-deployment-validation-operator"}, "namespace": "openshift-deployment-validation-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "91913dd5-e469-47bd-a73a-aaf1f6b6ee52", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-deployment-validation-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-dns-operator"}, "namespace": "openshift-dns-operator", "name": "openshift-service-ca.crt", "reconcileID": "99c6a92f-3052-4567-9a95-f1f34daf0bcf", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-dns-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-host-network"}, "namespace": "openshift-host-network", "name": "openshift-service-ca.crt", "reconcileID": "9fc03ebd-ae78-4598-b505-10c1a1f1b584", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-host-network"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"workload-variant-autoscaler-system"}, "namespace": "workload-variant-autoscaler-system", "name": "odh-trusted-ca-bundle", "reconcileID": "cf2b9c8c-8637-4831-8bce-51eb526c928c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "workload-variant-autoscaler-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-8ed53301"}, "namespace": "ai-tenant-e2e-aigw-8ed53301", "name": "openshift-service-ca.crt", "reconcileID": "e1de5e6d-a884-4998-b89e-cc49ccdd63de", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-8ed53301"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"kyverno"}, "namespace": "kyverno", "name": "odh-trusted-ca-bundle", "reconcileID": "ed3e1a27-43d1-4b0b-85ea-1a2b7cac515c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "kyverno"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-config"}, "namespace": "openshift-config", "name": "openshift-service-ca.crt", "reconcileID": "56c63e30-55a5-49aa-8354-178abe50fc8d", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-config"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-etcd"}, "namespace": "openshift-etcd", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4a7f1961-d12b-4361-a34c-32448b29424b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-etcd"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-88ae976f"}, "namespace": "ai-tenant-e2e-aigw-88ae976f", "name": "openshift-service-ca.crt", "reconcileID": "d2cb20a8-900b-46c2-8758-cec887a12de1", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-88ae976f"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"open-cluster-management-2r2qdb90ltvhjc43lp6iqc6aqb7bgjda"}, "namespace": "open-cluster-management-2r2qdb90ltvhjc43lp6iqc6aqb7bgjda", "name": "openshift-service-ca.crt", "reconcileID": "341b53a5-8c95-4aa6-bb18-8af621bd7882", "ConfigMap": "openshift-service-ca.crt", "namespace": "open-cluster-management-2r2qdb90ltvhjc43lp6iqc6aqb7bgjda"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-srep"}, "namespace": "openshift-backplane-srep", "name": "openshift-service-ca.crt", "reconcileID": "df8c9982-a8b3-4e89-b574-f8ee1498e86e", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-srep"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-infra"}, "namespace": "openshift-infra", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "61120953-d9e9-40b2-8cf3-cafec349b09b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-infra"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-ovn-kubernetes"}, "namespace": "openshift-ovn-kubernetes", "name": "openshift-service-ca.crt", "reconcileID": "0e784b81-98cc-4988-abbf-a9234004b303", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-ovn-kubernetes"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-277ae597"}, "namespace": "ai-tenant-e2e-aigw-277ae597", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "098334ff-cc6b-40b6-b1da-ed21a6028d90", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-277ae597"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-e072fffe"}, "namespace": "ai-tenant-e2e-aigw-e072fffe", "name": "openshift-service-ca.crt", "reconcileID": "54c290da-fce6-48a4-a12e-a9eff32fbdc2", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-e072fffe"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-system"}, "namespace": "kube-system", "name": "openshift-service-ca.crt", "reconcileID": "decc86a5-3a9e-494d-9dc9-77dd85e081c3", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-system"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-apiserver-operator"}, "namespace": "openshift-apiserver-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d515fa71-c993-4d35-ac5e-ff93fed2cbb2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-apiserver-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-version"}, "namespace": "openshift-cluster-version", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5daa8b80-97f0-4261-a804-41d954b9d3e7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-version"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-keda"}, "namespace": "openshift-keda", "name": "openshift-service-ca.crt", "reconcileID": "741a2fbd-7ea7-4c53-9216-4ff96e5935ba", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-keda"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-97xps"}, "namespace": "openshift-must-gather-97xps", "name": "openshift-service-ca.crt", "reconcileID": "9ad25ba6-37ae-41a8-a2f5-1ee5b40daf71", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-97xps"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "odh-trusted-ca-bundle", "reconcileID": "e1dc5cb3-98f4-45b3-beba-5130396cbbaf", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "redhat-ods-applications"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-af5eb82e"}, "namespace": "ai-tenant-e2e-aigw-af5eb82e", "name": "openshift-service-ca.crt", "reconcileID": "14411073-3b94-41d3-9e17-e2cbbfce0141", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-af5eb82e"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-evalhub-mcp-deployment"}, "namespace": "test-evalhub-mcp-deployment", "name": "openshift-service-ca.crt", "reconcileID": "8f31980d-7978-47f0-a70a-618b05be69b6", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-evalhub-mcp-deployment"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-console-operator"}, "namespace": "openshift-console-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4723221a-1f7d-40af-82b9-754cc23bceb5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-console-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-node"}, "namespace": "openshift-node", "name": "openshift-service-ca.crt", "reconcileID": "48102721-9bed-4979-b563-440a34079b82", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-node"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-service-ca"}, "namespace": "openshift-service-ca", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0209e747-cb04-486a-949f-94a2445b4423", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-service-ca"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"open-cluster-management-2r2qdb90ltvhjc43lp6iqc6aqb7bgjda"}, "namespace": "open-cluster-management-2r2qdb90ltvhjc43lp6iqc6aqb7bgjda", "name": "odh-trusted-ca-bundle", "reconcileID": "21dc3fdc-223b-4c4d-9023-1535ecc8f03e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "open-cluster-management-2r2qdb90ltvhjc43lp6iqc6aqb7bgjda"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-srep-ro"}, "namespace": "openshift-backplane-srep-ro", "name": "openshift-service-ca.crt", "reconcileID": "e76b8860-71c6-4db9-b0f3-cea4a843a13a", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-srep-ro"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cluster-samples-operator"}, "namespace": "openshift-cluster-samples-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "06df1e89-eb13-49d3-8c1c-abb822f53755", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cluster-samples-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-console"}, "namespace": "openshift-console", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2bc070df-039b-4cea-bde2-ffc720827e15", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-console"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-apiserver-operator"}, "namespace": "openshift-kube-apiserver-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d48d0f0b-6e99-4f2a-b19c-f59ec5ad4e37", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-apiserver-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kueue-managed-d989jnk35hkc74rp4pag"}, "namespace": "test-kueue-managed-d989jnk35hkc74rp4pag", "name": "openshift-service-ca.crt", "reconcileID": "bd3ecb9b-1971-445b-9140-fa12a53fde31", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kueue-managed-d989jnk35hkc74rp4pag"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"dedicated-admin"}, "namespace": "dedicated-admin", "name": "openshift-service-ca.crt", "reconcileID": "1f1f2a8a-4524-4e92-8ca1-4ad6bf665580", "ConfigMap": "openshift-service-ca.crt", "namespace": "dedicated-admin"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-console"}, "namespace": "openshift-console", "name": "openshift-service-ca.crt", "reconcileID": "52efaaa3-a9d3-41a6-9bdc-4f065f8a727b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-console"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-customer-monitoring"}, "namespace": "openshift-customer-monitoring", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "63f0e8f9-7ad9-49d7-b618-a0a844c84f23", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-customer-monitoring"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-storage-version-migrator-operator"}, "namespace": "openshift-kube-storage-version-migrator-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b684c61c-6757-4cbf-8b4d-2af2b063a366", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-storage-version-migrator-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kube-storage-version-migrator-operator"}, "namespace": "openshift-kube-storage-version-migrator-operator", "name": "openshift-service-ca.crt", "reconcileID": "04db9925-5343-403b-8801-8d4589dae8dc", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kube-storage-version-migrator-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-ogx-vector-stores-64f0"}, "namespace": "test-ogx-vector-stores-64f0", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ea0c1ec0-1ffb-475a-b314-64d9f67dd772", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-ogx-vector-stores-64f0"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-e072fffe"}, "namespace": "ai-tenant-e2e-aigw-e072fffe", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b1642678-2251-41e0-8606-313af56e6656", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-e072fffe"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-network-console"}, "namespace": "openshift-network-console", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "5b8b711d-1d20-4f17-90cf-c30580679cfa", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-network-console"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"cert-manager-operator"}, "namespace": "cert-manager-operator", "name": "odh-trusted-ca-bundle", "reconcileID": "7fb6c98e-3c1c-4bc8-8a30-aec2f23e7b08", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "cert-manager-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-backplane-mobb"}, "namespace": "openshift-backplane-mobb", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b6840f3a-2c28-409a-9887-cf34152b0d21", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-backplane-mobb"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "8e438501-7d7c-42b0-9cd2-1ca347d3dafc", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-a124ef78"}, "namespace": "ai-tenant-e2e-aigw-a124ef78", "name": "odh-trusted-ca-bundle", "reconcileID": "830972af-3fd6-429b-a100-ca08a367e03d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-a124ef78"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"dedicated-admin"}, "namespace": "dedicated-admin", "name": "odh-trusted-ca-bundle", "reconcileID": "d58c6e1f-86a7-4244-95e8-80b702ba9cef", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "dedicated-admin"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-kueue-operator"}, "namespace": "openshift-kueue-operator", "name": "openshift-service-ca.crt", "reconcileID": "1a3cfbd5-8146-49c8-a934-b3c5c7e23807", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-kueue-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-evalhub-tenant-a-testevalhubmcpdeployment-1b7577"}, "namespace": "test-evalhub-tenant-a-testevalhubmcpdeployment-1b7577", "name": "odh-trusted-ca-bundle", "reconcileID": "5ceb9c11-8454-482c-baee-bb717cff3427", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-evalhub-tenant-a-testevalhubmcpdeployment-1b7577"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "openshift-service-ca.crt", "reconcileID": "6609e031-ad79-4d28-9a5c-a67cd598457b", "ConfigMap": "openshift-service-ca.crt", "namespace": "rhods-notebooks"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-lws-operator"}, "namespace": "openshift-lws-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0c7cd902-b8f3-47aa-98e1-3f93eafa1c8d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-lws-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane"}, "namespace": "openshift-backplane", "name": "openshift-service-ca.crt", "reconcileID": "ffe18fc7-2c41-4cf6-8463-66fd2be47ec3", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-must-gather-4tx7s"}, "namespace": "openshift-must-gather-4tx7s", "name": "openshift-service-ca.crt", "reconcileID": "703bbc6d-359d-47f4-827b-e65a4b98362b", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-must-gather-4tx7s"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"oidc"}, "namespace": "oidc", "name": "openshift-service-ca.crt", "reconcileID": "3ab9a8ee-3050-4108-83bf-3d05f0dfb481", "ConfigMap": "openshift-service-ca.crt", "namespace": "oidc"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-backplane-cse"}, "namespace": "openshift-backplane-cse", "name": "openshift-service-ca.crt", "reconcileID": "b56a5150-81ec-4c1b-a848-11bac618718c", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-backplane-cse"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-jobset-operator"}, "namespace": "openshift-jobset-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "7e1abae6-286a-4661-9090-d1a6fe6a29cb", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-jobset-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-machine-api"}, "namespace": "openshift-machine-api", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3b9e914b-5fe8-4b64-a4d1-44100deb6695", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-machine-api"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-service-ca-operator"}, "namespace": "openshift-service-ca-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fbc5e8f1-8b17-402a-a64c-4ce0addfa01d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-service-ca-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kyverno"}, "namespace": "kyverno", "name": "openshift-service-ca.crt", "reconcileID": "28fa6d07-1f01-4afa-85cb-ad54ce027e15", "ConfigMap": "openshift-service-ca.crt", "namespace": "kyverno"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"openshift-dns"}, "namespace": "openshift-dns", "name": "openshift-service-ca.crt", "reconcileID": "214faa47-aa50-4b92-906b-23e2c167a031", "ConfigMap": "openshift-service-ca.crt", "namespace": "openshift-dns"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-e600492d"}, "namespace": "ai-tenant-e2e-aigw-e600492d", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "da877e58-ef36-4eaa-bccc-5c18a7c3bd49", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-e600492d"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-cloud-credential-operator"}, "namespace": "openshift-cloud-credential-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "0fc05b0d-2994-4ceb-bcac-a8dc57b8a1ee", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-cloud-credential-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-dns-operator"}, "namespace": "openshift-dns-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a51f4bd0-c94d-42d5-8270-0e2a4e2548ee", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-dns-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kueue-operator"}, "namespace": "openshift-kueue-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "d702876d-65c5-4473-813c-285e94fb3a43", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kueue-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-multus"}, "namespace": "openshift-multus", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "018117c5-8f4a-4b14-b05c-c84e9497aa8d", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-multus"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-operators"}, "namespace": "openshift-operators", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "1acc0d12-3448-4dbd-a9f6-6311eb4b6151", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-operators"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"kube-public"}, "namespace": "kube-public", "name": "openshift-service-ca.crt", "reconcileID": "5af09d9e-05d8-4d22-a9e3-81ae3e9150cc", "ConfigMap": "openshift-service-ca.crt", "namespace": "kube-public"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"openshift-kube-controller-manager-operator"}, "namespace": "openshift-kube-controller-manager-operator", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "4dbc0ff0-ba93-4bab-9215-662e6dbe04a2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "openshift-kube-controller-manager-operator"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-e600492d"}, "namespace": "ai-tenant-e2e-aigw-e600492d", "name": "openshift-service-ca.crt", "reconcileID": "a27143ab-1b29-4342-b836-067618c9d7b4", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-e600492d"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenants"}, "namespace": "ai-tenants", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c06d0dd2-47ff-46ac-85c3-622e6cb7ae12", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenants"} 2026-07-12T09:39:43Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "Namespace": "rhoai-model-registries"} 2026-07-12T09:39:43Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-standard-cpu {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "bed5787f-3d4d-4700-b2e2-1e0695d9317f", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "update": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "ConfigMap:": "opt-125m-standard-cpu-metrics-dashboard"} 2026-07-12T09:39:43Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-standard-cpu", "name": "inference-prometheus-auth"} 2026-07-12T09:39:43Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "autoscaling": null} 2026-07-12T09:39:43Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:39:43Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "Namespace": "rhoai-model-registries"} 2026-07-12T09:39:43Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-standard-cpu {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-standard-cpu","namespace":"opt-125m-standard-cpu"}, "namespace": "opt-125m-standard-cpu", "name": "opt-125m-standard-cpu", "reconcileID": "863260b2-7398-4b2d-aaa1-e4addfd3da1d", "InferenceService": "opt-125m-standard-cpu", "namespace": "opt-125m-standard-cpu"} 2026-07-12T09:45:54Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "2126aa8c-97a3-4701-b791-fb38d4f44db4", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "74d14dd2-c04c-4075-a988-05af2e743d01", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:45:54Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "74d14dd2-c04c-4075-a988-05af2e743d01", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "6850b057-26ce-4ce1-a21c-cef9e9cfa355", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:45:54Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "6850b057-26ce-4ce1-a21c-cef9e9cfa355", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "2b1aa356-b866-4b85-8acb-471dc6967bed", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "3c255745-f3bb-4fc8-b1e4-e695ab91b1f7", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "812bf629-aa71-46b5-9069-220c5c5a532a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:45:54Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "812bf629-aa71-46b5-9069-220c5c5a532a", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f1ab123b-476c-47b6-b616-cca8ef734d50", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z INFO Reconcile event triggered by serving runtime: http-mnist-runtime 2026-07-12T09:45:54Z INFO No InferenceServices found for Serving Runtime: http-mnist-runtime 2026-07-12T09:45:54Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z INFO Multi Node reconciling. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z INFO Creating Storage Config Secret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "ci-bucket-secret", "reconcileID": "facc3fd7-e870-492e-a767-8bb889e22db0", "Secret": "ci-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z INFO Reconcile event triggered by Secret: ci-bucket-secret 2026-07-12T09:45:54Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:45:54Z INFO Reconciling Storage Config Secret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"models-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "models-bucket-secret", "reconcileID": "a06838f5-556d-4a51-838a-8fdfdfe7be4f", "Secret": "models-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z INFO Reconcile event triggered by Secret: models-bucket-secret 2026-07-12T09:45:54Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:45:54Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:45:54Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "http-onnx", "name": "http-onnx"} 2026-07-12T09:45:54Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx"} 2026-07-12T09:45:54Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:45:54Z INFO inferenceservice-resource Namespace is not protected {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx"} 2026-07-12T09:45:54Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator"} 2026-07-12T09:45:54Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z ERROR Failed to fetch service for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "InferenceService": "http-onnx"} github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing /workspace/internal/controller/serving/inferenceservice_controller.go:164 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/inferenceservice_controller.go:173 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:45:54Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-metrics"} 2026-07-12T09:45:54Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-metrics"} 2026-07-12T09:45:54Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-metrics-dashboard"} 2026-07-12T09:45:54Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:54Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:54Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:54Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a6e5eeb3-008a-4a49-b22a-e9779c11da0b", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx"} 2026-07-12T09:45:55Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:45:55Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:55Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "Namespace": "rhoai-model-registries"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "4b832691-e830-4d62-ba7b-1023c0c7a10f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:45:55Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:45:55Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:55Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "Namespace": "rhoai-model-registries"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "21050ac1-b61c-472f-a901-49e331f1f215", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:45:55Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:45:55Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:55Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "Namespace": "rhoai-model-registries"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "209fd5a5-2c87-45ef-8085-a098d19eb6e8", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:45:55Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:45:55Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:55Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:55Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "Namespace": "rhoai-model-registries"} 2026-07-12T09:45:55Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d516583-7873-4829-8d6f-9cc497247c6f", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:45:56Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:45:56Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:56Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:56Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:56Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:56Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "Namespace": "rhoai-model-registries"} 2026-07-12T09:45:56Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "cd85d390-9b8c-4fbd-9c15-a1e5a6278ef6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:56Z INFO Multi Node reconciled successfully. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:45:58Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:45:58Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:58Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:58Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:45:58Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:45:58Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:45:58Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "Namespace": "rhoai-model-registries"} 2026-07-12T09:45:58Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "64548e6f-d7ab-44d1-979c-702e735944d6", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:07Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:07Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:07Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:07Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:07Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:07Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "de4c9494-97fe-4407-aa8a-c1d80e28a894", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "delete": "test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator"} 2026-07-12T09:46:17Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:17Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "66a97a4a-89a7-4af0-9874-7fe3837579f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:17Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "62d8f7da-e976-459e-99df-dfd83050290a", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:17Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "506f1cb5-9481-40d5-b0d6-9616c14b2915", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:17Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c1b8ee6e-f06a-4fc6-b284-f15aed6b13b0", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:17Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:17Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:17Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:17Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9d8a0601-7c27-4891-92ce-ebfb937b30cb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:18Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:18Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:18Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:18Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:18Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:18Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "a0277fdf-61ad-4a74-a719-ad9a3f201e41", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:20Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:20Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:20Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:20Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:20Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:20Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:20Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:20Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:20Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:20Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:20Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "c6ae9e06-26e1-448a-a42d-3aa1d6da2a1e", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "create": "test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator"} 2026-07-12T09:46:24Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:24Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:24Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:24Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5b272606-4b07-488d-9897-a2dd962d6876", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:24Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:24Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:24Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "e0e9db48-c98f-4952-8d82-c764178c8240", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:24Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:24Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:24Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "5a0cfe4a-f602-43ab-a77e-c6d600482fed", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx"} 2026-07-12T09:46:24Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-metrics-dashboard"} 2026-07-12T09:46:24Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:24Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:24Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:24Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:24Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "f5229e74-4c5d-4a3e-9fc6-ac3d8dae73f3", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "http-onnx-2", "name": "http-onnx-2"} 2026-07-12T09:46:31Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx-2"} 2026-07-12T09:46:31Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:46:31Z INFO inferenceservice-resource Namespace is not protected {"namespace": "test-kserve-raw-token-authentication", "isvc": "http-onnx-2"} 2026-07-12T09:46:31Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "test-kserve-raw-token-authentication-models-bucket-sa-2-auth-delegator"} 2026-07-12T09:46:31Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z ERROR Failed to fetch service for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "InferenceService": "http-onnx-2"} github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing /workspace/internal/controller/serving/inferenceservice_controller.go:164 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/inferenceservice_controller.go:173 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:46:31Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-2-metrics"} 2026-07-12T09:46:31Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-2-metrics"} 2026-07-12T09:46:31Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:31Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "45c00d59-6a4b-4216-bf4d-1a1386560d0b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "create": "http-onnx-2"} 2026-07-12T09:46:31Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:46:31Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:31Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "e80dec9f-6a47-4481-a850-45dfff8b946c", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:46:31Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:46:31Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:31Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3723277f-5525-4784-904b-dd0f4d058eaa", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:46:31Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:46:31Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:31Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:31Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:31Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "30d23d67-b45f-4dc9-9cf5-7f5ea7eca91a", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:46:32Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:46:32Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:32Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:32Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:32Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:32Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:32Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:32Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "26c804d8-869c-4fa8-b1a9-10de62314435", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:46:34Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:46:34Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:34Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:34Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:34Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:34Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:34Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:34Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:34Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:34Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:34Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:34Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "5851671f-e28f-459e-9260-6e0084a74944", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG Successfully fetch deployed ClusterRoleBinding {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "update": "http-onnx-2"} 2026-07-12T09:46:43Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "ConfigMap:": "http-onnx-2-metrics-dashboard"} 2026-07-12T09:46:43Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:43Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:43Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:43Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:43Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:43Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:43Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:43Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3c2bcd99-fb14-4f12-9a8f-375b2a966511", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2"} 2026-07-12T09:46:53Z DEBUG Deleting ClusterRoleBinding test-kserve-raw-token-authentication-models-bucket-sa-2-auth-delegator in namespace test-kserve-raw-token-authentication {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:53Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-2 {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "7ef518af-e706-44b8-9fd6-58592ddea02f", "error": "InferenceService.serving.kserve.io \"http-onnx-2\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:46:53Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2"} 2026-07-12T09:46:53Z DEBUG Deleting ClusterRoleBinding test-kserve-raw-token-authentication-models-bucket-sa-2-auth-delegator in namespace test-kserve-raw-token-authentication {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:53Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:53Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:53Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "de17903d-9097-40a0-aac3-ddda5a8dbb50", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "213c704a-e5dd-42ae-81fe-d7e4a347de1b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "213c704a-e5dd-42ae-81fe-d7e4a347de1b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:53Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "213c704a-e5dd-42ae-81fe-d7e4a347de1b", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3f560a05-0929-49e0-b218-1ee8cb0fe279", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:53Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3f560a05-0929-49e0-b218-1ee8cb0fe279", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:53Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-2","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx-2", "reconcileID": "3f560a05-0929-49e0-b218-1ee8cb0fe279", "InferenceService": "http-onnx-2", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx"} 2026-07-12T09:46:54Z DEBUG Deleting ClusterRoleBinding test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator in namespace test-kserve-raw-token-authentication {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:54Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:54Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14aef3de-329c-4e86-ba04-cc3703959dd5", "error": "InferenceService.serving.kserve.io \"http-onnx\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:46:54Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx"} 2026-07-12T09:46:54Z DEBUG Deleting ClusterRoleBinding test-kserve-raw-token-authentication-models-bucket-sa-auth-delegator in namespace test-kserve-raw-token-authentication {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-kserve-raw-token-authentication", "name": "inference-prometheus-auth"} 2026-07-12T09:46:54Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "autoscaling": null} 2026-07-12T09:46:54Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:54Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:55Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "14d922e3-05f2-461c-99ed-def14e4e1f23", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9cbdf614-3a20-4c54-aab5-f85f2ca506fb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9cbdf614-3a20-4c54-aab5-f85f2ca506fb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9cbdf614-3a20-4c54-aab5-f85f2ca506fb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9cbdf614-3a20-4c54-aab5-f85f2ca506fb", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:55Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "9cbdf614-3a20-4c54-aab5-f85f2ca506fb", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "735b7c64-a9a0-4329-8732-7ae1998c9abd", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "735b7c64-a9a0-4329-8732-7ae1998c9abd", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "735b7c64-a9a0-4329-8732-7ae1998c9abd", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "735b7c64-a9a0-4329-8732-7ae1998c9abd", "Namespace": "rhoai-model-registries"} 2026-07-12T09:46:55Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "http-onnx", "reconcileID": "735b7c64-a9a0-4329-8732-7ae1998c9abd", "InferenceService": "http-onnx", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"models-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "models-bucket-secret", "reconcileID": "961a2640-1daf-4233-9aed-7ec746dd9500", "Secret": "models-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO Reconciling Storage Config Secret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"models-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "models-bucket-secret", "reconcileID": "961a2640-1daf-4233-9aed-7ec746dd9500", "Secret": "models-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:55Z INFO Reconcile event triggered by Secret: models-bucket-secret 2026-07-12T09:46:55Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:46:55Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:46:56Z INFO Reconcile event triggered by Secret: ci-bucket-secret 2026-07-12T09:46:56Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "ci-bucket-secret", "reconcileID": "917b082c-8082-4fa9-aa02-b03a28bd34fe", "Secret": "ci-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:56Z INFO No data connections found in namespace {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "ci-bucket-secret", "reconcileID": "917b082c-8082-4fa9-aa02-b03a28bd34fe", "Secret": "ci-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:56Z INFO Deleting StorageSecret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-secret","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "ci-bucket-secret", "reconcileID": "917b082c-8082-4fa9-aa02-b03a28bd34fe", "Secret": "ci-bucket-secret", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:56Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:46:56Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"storage-config","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "storage-config", "reconcileID": "e114a1b4-3404-4f53-9d75-7bc106fc4c18", "Secret": "storage-config", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:56Z INFO No data connections found in namespace {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"storage-config","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "storage-config", "reconcileID": "e114a1b4-3404-4f53-9d75-7bc106fc4c18", "Secret": "storage-config", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:56Z INFO Reconcile event triggered by serving runtime: http-mnist-runtime 2026-07-12T09:46:56Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:56Z INFO No InferenceServices found for Serving Runtime: http-mnist-runtime 2026-07-12T09:46:56Z INFO Multi Node reconciling. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:46:57Z INFO Multi Node reconciled successfully. {"ResourceName": "http-mnist-runtime", "Namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:03Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e5b21ad7-1139-40e0-9366-0ab905080781", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:03Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e5b21ad7-1139-40e0-9366-0ab905080781", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:47:03Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e5b21ad7-1139-40e0-9366-0ab905080781", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-kserve-raw-token-authentication because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:47:03Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b0405930-ffc0-41f8-9f35-5e272ec7c8da", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:03Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b0405930-ffc0-41f8-9f35-5e272ec7c8da", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:47:03Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b0405930-ffc0-41f8-9f35-5e272ec7c8da", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-kserve-raw-token-authentication because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:47:03Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "24156d4e-5dd1-4be8-a0ae-f50329fe29d3", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:03Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "24156d4e-5dd1-4be8-a0ae-f50329fe29d3", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:47:03Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "24156d4e-5dd1-4be8-a0ae-f50329fe29d3", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-kserve-raw-token-authentication because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:47:03Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "d9f536df-97fc-4bd5-9876-e4a838e7af74", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:03Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "openshift-service-ca.crt", "reconcileID": "d9f536df-97fc-4bd5-9876-e4a838e7af74", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:03Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "6b401c5f-4a10-49fc-b4b3-36ae3cb7fb48", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:03Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-trusted-ca-bundle", "reconcileID": "6b401c5f-4a10-49fc-b4b3-36ae3cb7fb48", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:03Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "00565030-87ae-4e58-8885-4f7e8ffe9bc1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:03Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-kserve-raw-token-authentication"}, "namespace": "test-kserve-raw-token-authentication", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "00565030-87ae-4e58-8885-4f7e8ffe9bc1", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-kserve-raw-token-authentication"} 2026-07-12T09:47:05Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "a257a5c0-64aa-412a-96f4-d2ec855886a1", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "5f483739-abf0-4bc4-bf8b-073dac4b4504", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:47:05Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "5f483739-abf0-4bc4-bf8b-073dac4b4504", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "31f49448-a9eb-4d5d-a7ac-3415938cd468", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:47:05Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "31f49448-a9eb-4d5d-a7ac-3415938cd468", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "eba34886-147a-42ad-9587-0925ec340775", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "271dad3a-fd72-41c6-b843-22289e452860", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "d1314fa6-c1ad-49cf-b0f7-97410e2e3490", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:47:05Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "d1314fa6-c1ad-49cf-b0f7-97410e2e3490", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "239a3576-1d7f-4b36-84d3-449334378659", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z INFO Reconcile event triggered by serving runtime: http-mnist-runtime 2026-07-12T09:47:05Z INFO No InferenceServices found for Serving Runtime: http-mnist-runtime 2026-07-12T09:47:05Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z INFO Multi Node reconciling. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z INFO Creating Storage Config Secret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-unprivileged","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "ci-bucket-unprivileged", "reconcileID": "6e63d4b3-f767-4eb8-a5a8-21c6c5468cde", "Secret": "ci-bucket-unprivileged", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z INFO Reconcile event triggered by Secret: ci-bucket-unprivileged 2026-07-12T09:47:05Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:47:05Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "http-onnx-raw", "name": "http-onnx-raw"} 2026-07-12T09:47:05Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "test-non-admin-raw", "isvc": "http-onnx-raw"} 2026-07-12T09:47:05Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "test-non-admin-raw", "isvc": "http-onnx-raw", "namespace": "test-non-admin-raw", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:47:05Z INFO inferenceservice-resource Namespace is not protected {"namespace": "test-non-admin-raw", "isvc": "http-onnx-raw"} 2026-07-12T09:47:05Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:05Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "create": "http-onnx-raw-metrics"} 2026-07-12T09:47:06Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "create": "http-onnx-raw-metrics"} 2026-07-12T09:47:06Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "create": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:06Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:06Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "66e32653-32ac-44db-9f1d-e5780df88811", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:47:06Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:06Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:06Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "47ee8906-7aac-46d1-83be-802167e1703f", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:47:06Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:06Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:06Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "b5d2acb8-60cb-415c-9939-c499cfe3e415", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:47:06Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:06Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:06Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:06Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:06Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f6b0dbe4-29ee-43f8-b201-d5d523aa1ec8", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:47:07Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:07Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:07Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:07Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:07Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "8cdfef8f-4d8a-4dd8-aec5-9af9dd635095", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:47:08Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:08Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:08Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:08Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:08Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:08Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "3c4a7081-d94d-4508-b12c-eae300d06a95", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:09Z INFO Multi Node reconciled successfully. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "ConfigMap:": "http-onnx-raw-metrics-dashboard"} 2026-07-12T09:47:16Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:16Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:16Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:16Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:16Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:16Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:16Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:16Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:16Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:16Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6e0416f6-f653-463d-8ab5-82673e905ef7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "name": "http-onnx-raw"} 2026-07-12T09:47:18Z DEBUG Deleting ClusterRoleBinding test-non-admin-raw-default-auth-delegator in namespace test-non-admin-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:18Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: http-onnx-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f93d83e0-a289-44dd-bc19-83697c84acb6", "error": "InferenceService.serving.kserve.io \"http-onnx-raw\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:47:18Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "name": "http-onnx-raw"} 2026-07-12T09:47:18Z DEBUG Deleting ClusterRoleBinding test-non-admin-raw-default-auth-delegator in namespace test-non-admin-raw {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "test-non-admin-raw", "name": "inference-prometheus-auth"} 2026-07-12T09:47:18Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "autoscaling": null} 2026-07-12T09:47:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:18Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "6f28384c-2d79-4150-b3c1-7338f7c2f159", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f663e340-959d-44b3-9cc0-3784cec104a7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f663e340-959d-44b3-9cc0-3784cec104a7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f663e340-959d-44b3-9cc0-3784cec104a7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f663e340-959d-44b3-9cc0-3784cec104a7", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:18Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "f663e340-959d-44b3-9cc0-3784cec104a7", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "cd180b0f-f080-4ff2-a6f1-455a925cc33e", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "cd180b0f-f080-4ff2-a6f1-455a925cc33e", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "cd180b0f-f080-4ff2-a6f1-455a925cc33e", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:18Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "cd180b0f-f080-4ff2-a6f1-455a925cc33e", "Namespace": "rhoai-model-registries"} 2026-07-12T09:47:18Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"http-onnx-raw","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "http-onnx-raw", "reconcileID": "cd180b0f-f080-4ff2-a6f1-455a925cc33e", "InferenceService": "http-onnx-raw", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:19Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-unprivileged","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "ci-bucket-unprivileged", "reconcileID": "65a85724-fe81-4b42-8cb8-166d710a6f6b", "Secret": "ci-bucket-unprivileged", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:19Z INFO No data connections found in namespace {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-unprivileged","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "ci-bucket-unprivileged", "reconcileID": "65a85724-fe81-4b42-8cb8-166d710a6f6b", "Secret": "ci-bucket-unprivileged", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:19Z INFO Deleting StorageSecret {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"ci-bucket-unprivileged","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "ci-bucket-unprivileged", "reconcileID": "65a85724-fe81-4b42-8cb8-166d710a6f6b", "Secret": "ci-bucket-unprivileged", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:19Z INFO Reconcile event triggered by Secret: ci-bucket-unprivileged 2026-07-12T09:47:19Z INFO Reconcile event triggered by Secret: storage-config 2026-07-12T09:47:19Z INFO Data Connection not found {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"storage-config","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "storage-config", "reconcileID": "10a809e6-efe6-40ef-85b4-14096ef31be6", "Secret": "storage-config", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:19Z INFO No data connections found in namespace {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"storage-config","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "storage-config", "reconcileID": "10a809e6-efe6-40ef-85b4-14096ef31be6", "Secret": "storage-config", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:19Z INFO Reconcile event triggered by serving runtime: http-mnist-runtime 2026-07-12T09:47:19Z INFO No InferenceServices found for Serving Runtime: http-mnist-runtime 2026-07-12T09:47:19Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:47:19Z INFO Multi Node reconciling. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:47:20Z INFO Multi Node reconciled successfully. {"ResourceName": "http-mnist-runtime", "Namespace": "test-non-admin-raw"} 2026-07-12T09:47:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b0467338-8dcd-4c6f-a627-d871cf85c34b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:27Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b0467338-8dcd-4c6f-a627-d871cf85c34b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:47:27Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "b0467338-8dcd-4c6f-a627-d871cf85c34b", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-non-admin-raw because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:47:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ffead73a-cd70-4a53-8450-fabcd3a3c00c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:27Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ffead73a-cd70-4a53-8450-fabcd3a3c00c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:47:27Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ffead73a-cd70-4a53-8450-fabcd3a3c00c", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-non-admin-raw because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:47:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "3dd35b35-9b19-4fc7-8462-43b861017531", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:27Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "3dd35b35-9b19-4fc7-8462-43b861017531", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:47:27Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "3dd35b35-9b19-4fc7-8462-43b861017531", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace test-non-admin-raw because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:47:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "a4fa7aee-7460-444f-be41-5cbd37f4eb72", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:27Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "openshift-service-ca.crt", "reconcileID": "a4fa7aee-7460-444f-be41-5cbd37f4eb72", "ConfigMap": "openshift-service-ca.crt", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "996a4c51-ffd8-45cc-9aeb-ad98d2c39779", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:27Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-trusted-ca-bundle", "reconcileID": "996a4c51-ffd8-45cc-9aeb-ad98d2c39779", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "382bb8af-cfd2-46a5-8bb6-9794e947544f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:47:27Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"test-non-admin-raw"}, "namespace": "test-non-admin-raw", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "382bb8af-cfd2-46a5-8bb6-9794e947544f", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "test-non-admin-raw"} 2026-07-12T09:48:06Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "6849c3f8-244c-419e-b2b2-8a91b2d802b4", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:06Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "9ee4ba5a-ac76-4016-b02c-3e37419a76d3", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:06Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "9ee4ba5a-ac76-4016-b02c-3e37419a76d3", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:06Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "772f1fb9-6d1d-46ce-b7d8-6cff95120236", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:06Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "772f1fb9-6d1d-46ce-b7d8-6cff95120236", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:06Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e8367c1c-a781-403c-936a-d548ba8f6001", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:06Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "bfd2eada-29ee-4b0b-91ab-a101a89df1d8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:06Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "bfd2eada-29ee-4b0b-91ab-a101a89df1d8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:06Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "80d10b0e-3453-442f-b9b0-441f6afc65a9", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Reconcile event triggered by serving runtime: mlserver-runtime 2026-07-12T09:48:07Z INFO No InferenceServices found for Serving Runtime: mlserver-runtime 2026-07-12T09:48:07Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Multi Node reconciling. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "sklearn-modelcar", "name": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "sklearn-modelcar", "isvc": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "sklearn-modelcar", "isvc": "sklearn-modelcar", "namespace": "sklearn-modelcar", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:48:07Z INFO inferenceservice-resource Namespace is not protected {"namespace": "sklearn-modelcar", "isvc": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "create": "sklearn-modelcar-metrics"} 2026-07-12T09:48:07Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "create": "sklearn-modelcar-metrics"} 2026-07-12T09:48:07Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "create": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:07Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:07Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "ace5c432-e1fe-4f7d-9b02-c618d08a8c4f", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "ConfigMap:": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:48:07Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:07Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:07Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "4e96e679-d80a-4196-926c-b39f7004e8e2", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "ConfigMap:": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:48:07Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:07Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:07Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2b015325-19c6-425f-93da-c7ae24a0dcb4", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "ConfigMap:": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:48:07Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:07Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:07Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:07Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:07Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "02583564-2e95-4217-b3e3-f63ebd04bb22", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:09Z INFO Multi Node reconciled successfully. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "ConfigMap:": "sklearn-modelcar-metrics-dashboard"} 2026-07-12T09:48:18Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:18Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "2d63e70e-57a3-4c4a-ad4d-b2e958526e7d", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "name": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Deleting ClusterRoleBinding sklearn-modelcar-default-auth-delegator in namespace sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "sklearn-modelcar", "name": "inference-prometheus-auth"} 2026-07-12T09:48:18Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "autoscaling": null} 2026-07-12T09:48:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:18Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "bf0171e5-8a1b-4696-9156-3160037fad9b", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "784c8a1b-20a0-4815-979c-048f64ee313c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:18Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "784c8a1b-20a0-4815-979c-048f64ee313c", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:18Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: sklearn-modelcar {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "784c8a1b-20a0-4815-979c-048f64ee313c", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "abb5c3dd-ec53-4cd0-a5c4-bfd8f07a4da7", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "abb5c3dd-ec53-4cd0-a5c4-bfd8f07a4da7", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "abb5c3dd-ec53-4cd0-a5c4-bfd8f07a4da7", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "abb5c3dd-ec53-4cd0-a5c4-bfd8f07a4da7", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:19Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "abb5c3dd-ec53-4cd0-a5c4-bfd8f07a4da7", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "053c3f45-e259-422d-ae05-6dc9c6840314", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "053c3f45-e259-422d-ae05-6dc9c6840314", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "053c3f45-e259-422d-ae05-6dc9c6840314", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "053c3f45-e259-422d-ae05-6dc9c6840314", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:19Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-modelcar","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "sklearn-modelcar", "reconcileID": "053c3f45-e259-422d-ae05-6dc9c6840314", "InferenceService": "sklearn-modelcar", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z INFO Multi Node reconciling. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:48:19Z INFO Reconcile event triggered by serving runtime: mlserver-runtime 2026-07-12T09:48:19Z INFO No InferenceServices found for Serving Runtime: mlserver-runtime 2026-07-12T09:48:20Z INFO Multi Node reconciled successfully. {"ResourceName": "mlserver-runtime", "Namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c55807ff-c704-4bb9-8a4d-2ce41fb4bee7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c55807ff-c704-4bb9-8a4d-2ce41fb4bee7", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:27Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c55807ff-c704-4bb9-8a4d-2ce41fb4bee7", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace sklearn-modelcar because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:48:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "def782e9-daf4-4db7-95fb-3b37480c2c6c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "def782e9-daf4-4db7-95fb-3b37480c2c6c", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:27Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "def782e9-daf4-4db7-95fb-3b37480c2c6c", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace sklearn-modelcar because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:48:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "f893059c-878e-4296-816b-596dc4e78b3e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "f893059c-878e-4296-816b-596dc4e78b3e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:27Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "f893059c-878e-4296-816b-596dc4e78b3e", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace sklearn-modelcar because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:48:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c95afa9b-7f79-43b4-8298-5008a5278ba3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c95afa9b-7f79-43b4-8298-5008a5278ba3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:27Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "c95afa9b-7f79-43b4-8298-5008a5278ba3", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace sklearn-modelcar because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:48:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "bd2bc149-6eac-4973-ae7a-0c30a1880258", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "openshift-service-ca.crt", "reconcileID": "bd2bc149-6eac-4973-ae7a-0c30a1880258", "ConfigMap": "openshift-service-ca.crt", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "f4acf9f6-0d9f-4dda-9162-b18de8e5adde", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-trusted-ca-bundle", "reconcileID": "f4acf9f6-0d9f-4dda-9162-b18de8e5adde", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fb0ad4e3-d922-4354-8804-d9c5bdd5fab0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:27Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"sklearn-modelcar"}, "namespace": "sklearn-modelcar", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "fb0ad4e3-d922-4354-8804-d9c5bdd5fab0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "sklearn-modelcar"} 2026-07-12T09:48:29Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "openshift-service-ca.crt", "reconcileID": "ea5430ab-7ebb-41b5-8d84-19668da8679b", "ConfigMap": "openshift-service-ca.crt", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "2240c903-cca5-4a93-807a-56331d9c546c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:29Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "2240c903-cca5-4a93-807a-56331d9c546c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "openshift-service-ca.crt", "reconcileID": "2e6bb0aa-9004-4b90-bff5-3b569cc3b876", "ConfigMap": "openshift-service-ca.crt", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "315b826b-5268-4e02-9736-3cbc2e31dac4", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "235e8f8b-7e72-43a1-b8ca-97c4f5864a2d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:29Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z INFO Multi Node reconciling. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z INFO Reconcile event triggered by serving runtime: openvino-runtime 2026-07-12T09:48:29Z INFO No InferenceServices found for Serving Runtime: openvino-runtime 2026-07-12T09:48:29Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "235e8f8b-7e72-43a1-b8ca-97c4f5864a2d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a6f26d30-9889-4972-9321-fe79fe98fcad", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "onnx", "name": "onnx"} 2026-07-12T09:48:29Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "onnx-standard-rest", "isvc": "onnx"} 2026-07-12T09:48:29Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "onnx-standard-rest", "isvc": "onnx", "namespace": "onnx-standard-rest", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:48:29Z INFO inferenceservice-resource Namespace is not protected {"namespace": "onnx-standard-rest", "isvc": "onnx"} 2026-07-12T09:48:29Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "create": "onnx-metrics"} 2026-07-12T09:48:29Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "create": "onnx-metrics"} 2026-07-12T09:48:29Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "create": "onnx-metrics-dashboard"} 2026-07-12T09:48:29Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:29Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:29Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:29Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:29Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:29Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:29Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:29Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:29Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "24c365d3-2bd1-4e8a-8bb6-d01bb44add05", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:29Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:48:30Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:30Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:30Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "f11e3063-7bc6-414a-a604-89081a739f87", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:48:30Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:30Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:30Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "6866a159-64e2-4fb8-acc8-e3311977c9eb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:48:30Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:30Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:30Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "51c95f6f-e5ab-4041-8341-672ec9572c2f", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:48:30Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:30Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:30Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:30Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "acfd8eac-f11d-4ebd-ac21-a854028b5a22", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:30Z INFO Multi Node reconciled successfully. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:48:32Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:32Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:32Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:32Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:32Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:32Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:32Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "cc8e968c-888f-4de6-b1c1-e3568f3326e6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:48:33Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:33Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:33Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:33Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:33Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "47a2778e-fb33-4c6e-8d56-0fcdb730c785", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "ConfigMap:": "onnx-metrics-dashboard"} 2026-07-12T09:48:41Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:41Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "2f6bf555-9d91-40a3-a9bc-9cc92068b4be", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "name": "onnx"} 2026-07-12T09:48:41Z DEBUG Deleting ClusterRoleBinding onnx-standard-rest-openvino-models-bucket-sa-auth-delegator in namespace onnx-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:41Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: onnx {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "d857cee8-11d7-4845-b33a-8304d0d7b5bb", "error": "InferenceService.serving.kserve.io \"onnx\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:48:41Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "name": "onnx"} 2026-07-12T09:48:41Z DEBUG Deleting ClusterRoleBinding onnx-standard-rest-openvino-models-bucket-sa-auth-delegator in namespace onnx-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "onnx-standard-rest", "name": "inference-prometheus-auth"} 2026-07-12T09:48:41Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "autoscaling": null} 2026-07-12T09:48:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:41Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "8e4414ca-6aa1-4251-a201-9c7d47966cc6", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "49623ee5-4674-4914-8ccd-47223ee243cc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "49623ee5-4674-4914-8ccd-47223ee243cc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "49623ee5-4674-4914-8ccd-47223ee243cc", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "49623ee5-4674-4914-8ccd-47223ee243cc", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:41Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "49623ee5-4674-4914-8ccd-47223ee243cc", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "1571a4ab-297b-4284-8250-b75db02eccc8", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "1571a4ab-297b-4284-8250-b75db02eccc8", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "1571a4ab-297b-4284-8250-b75db02eccc8", "InferenceService": "onnx", "namespace": "onnx-standard-rest", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:41Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "1571a4ab-297b-4284-8250-b75db02eccc8", "Namespace": "rhoai-model-registries"} 2026-07-12T09:48:41Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"onnx","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "onnx", "reconcileID": "1571a4ab-297b-4284-8250-b75db02eccc8", "InferenceService": "onnx", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:42Z INFO Reconcile event triggered by serving runtime: openvino-runtime 2026-07-12T09:48:42Z INFO No InferenceServices found for Serving Runtime: openvino-runtime 2026-07-12T09:48:42Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:48:42Z INFO Multi Node reconciling. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:48:45Z INFO Multi Node reconciled successfully. {"ResourceName": "openvino-runtime", "Namespace": "onnx-standard-rest"} 2026-07-12T09:48:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e9e21445-2698-4804-8311-a1d35b1ed70e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:49Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e9e21445-2698-4804-8311-a1d35b1ed70e", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:49Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e9e21445-2698-4804-8311-a1d35b1ed70e", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace onnx-standard-rest because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:48:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ad409601-e93c-4551-abb1-56ded513d3b0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:49Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ad409601-e93c-4551-abb1-56ded513d3b0", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:49Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ad409601-e93c-4551-abb1-56ded513d3b0", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace onnx-standard-rest because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:48:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "72b99cb1-57e2-4a78-bccb-ad738691ede4", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:49Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "72b99cb1-57e2-4a78-bccb-ad738691ede4", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:49Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "72b99cb1-57e2-4a78-bccb-ad738691ede4", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace onnx-standard-rest because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:48:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "openshift-service-ca.crt", "reconcileID": "bd556639-2f72-45fc-9a96-78c33aa809b0", "ConfigMap": "openshift-service-ca.crt", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:49Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "openshift-service-ca.crt", "reconcileID": "bd556639-2f72-45fc-9a96-78c33aa809b0", "ConfigMap": "openshift-service-ca.crt", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "066c355e-2d19-4cd5-a506-bb645e3d67c8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:49Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-trusted-ca-bundle", "reconcileID": "066c355e-2d19-4cd5-a506-bb645e3d67c8", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:49Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "88208c9a-367a-480f-96b7-8597455074f5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:49Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"onnx-standard-rest"}, "namespace": "onnx-standard-rest", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "88208c9a-367a-480f-96b7-8597455074f5", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "onnx-standard-rest"} 2026-07-12T09:48:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "1f2f6b74-7191-4b00-84e8-b5c92a69852a", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:48:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "9a6af953-b1ab-4c55-828f-dd2a61c54dcb", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:53Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "9a6af953-b1ab-4c55-828f-dd2a61c54dcb", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:48:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "18b4ae6d-b01a-401e-86da-ad3829fdfa37", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:53Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "18b4ae6d-b01a-401e-86da-ad3829fdfa37", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:48:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a3e0e435-f54c-4a58-bb12-f4369a4af505", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:48:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "91577d11-571f-49d0-ae31-356f231cb773", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:48:53Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "c5b7592c-e5f8-40b4-8958-9123bf8a9576", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:48:53Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "c5b7592c-e5f8-40b4-8958-9123bf8a9576", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:48:53Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "f6594c5b-ca2a-4dba-bae8-bb73cefe67a3", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:49:28Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "467df574-c017-4d5c-a1ad-463738330e60", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:49:28Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "467df574-c017-4d5c-a1ad-463738330e60", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:49:28Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "467df574-c017-4d5c-a1ad-463738330e60", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ovms-smoke because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:49:28Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "72e3749f-91e7-4e49-ba03-aecbad243827", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:49:28Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "72e3749f-91e7-4e49-ba03-aecbad243827", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:49:28Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "72e3749f-91e7-4e49-ba03-aecbad243827", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace ovms-smoke because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:49:28Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "f29cfea1-8c87-4d52-ad43-674090201bba", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:49:28Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-trusted-ca-bundle", "reconcileID": "f29cfea1-8c87-4d52-ad43-674090201bba", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:49:28Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "888b14df-84f2-456b-a09a-2ac3734d4e3c", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:49:28Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "openshift-service-ca.crt", "reconcileID": "888b14df-84f2-456b-a09a-2ac3734d4e3c", "ConfigMap": "openshift-service-ca.crt", "namespace": "ovms-smoke"} 2026-07-12T09:49:28Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dd0e562b-80ef-47e4-a426-1fc7f2b80857", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:49:28Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ovms-smoke"}, "namespace": "ovms-smoke", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "dd0e562b-80ef-47e4-a426-1fc7f2b80857", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ovms-smoke"} 2026-07-12T09:49:32Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "2cf5fc58-6c31-442d-a158-5873448d3ebf", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:49:32Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "c51c8cca-8b4a-4a18-b3c8-a58f029e965f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:49:32Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "c51c8cca-8b4a-4a18-b3c8-a58f029e965f", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:49:32Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "f853d702-f54c-48cc-b44e-fcea780be016", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:49:32Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "f853d702-f54c-48cc-b44e-fcea780be016", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:49:32Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "33cf2565-92f1-4e4e-b614-408ceec541ef", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:49:32Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "cba99ced-2162-422e-b749-6c4ab92bb4db", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:49:32Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "4c815884-7a3e-4195-9097-2fc1bbae9d6e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:49:32Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "4c815884-7a3e-4195-9097-2fc1bbae9d6e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:49:32Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "9463e27b-f09c-4bad-88f6-40dc976c9058", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:49:32Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:49:32Z INFO Multi Node reconciling. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:49:32Z INFO Reconcile event triggered by serving runtime: triton-rest-runtime 2026-07-12T09:49:32Z INFO No InferenceServices found for Serving Runtime: triton-rest-runtime 2026-07-12T09:49:33Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "tensorflow-standard-rest", "name": "tensorflow-standard-rest"} 2026-07-12T09:49:33Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "tensorflow-standard", "isvc": "tensorflow-standard-rest"} 2026-07-12T09:49:33Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "tensorflow-standard", "isvc": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:49:33Z INFO inferenceservice-resource Namespace is not protected {"namespace": "tensorflow-standard", "isvc": "tensorflow-standard-rest"} 2026-07-12T09:49:33Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "create": "tensorflow-standard-rest-metrics"} 2026-07-12T09:49:33Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "create": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:33Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:33Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "e8b47c2b-be46-4254-8c38-7dc98eb99546", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:49:33Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:33Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:33Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4983ce89-b006-45c9-8692-fd6dcb685750", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:49:33Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:33Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:33Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "4b2fd26a-ac0f-4520-ac19-e68e88fc74df", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:49:33Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:33Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:33Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:33Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:33Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "92038e70-4388-4058-be17-649702e81d95", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:49:35Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:35Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:35Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:35Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:35Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:35Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:35Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:35Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:35Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5475465-04d1-49a3-85fb-3b8c8803e6b0", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:35Z INFO Multi Node reconciled successfully. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:49:37Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:37Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:37Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:37Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:37Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:37Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:37Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "a5234655-f621-41bb-a851-2f92ed302e13", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z INFO InferenceService does not have label 'networking.kserve.io/visibility' annotation set to 'exposed'. Skipping route creation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG No Prometheus annotations on ServingRuntime, skipping creation of metrics resources {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "ConfigMap:": "tensorflow-standard-rest-metrics-dashboard"} 2026-07-12T09:49:46Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:46Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:46Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:46Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:46Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:46Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:46Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "c356166a-3c20-411c-9b1d-8c971558d3cf", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest"} 2026-07-12T09:49:57Z DEBUG Deleting ClusterRoleBinding tensorflow-standard-triton-models-bucket-sa-auth-delegator in namespace tensorflow-standard {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:57Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: tensorflow-standard-rest {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z ERROR Reconciler error {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "07e69126-3cab-4164-91f7-53ab05cf65fd", "error": "InferenceService.serving.kserve.io \"tensorflow-standard-rest\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"inferenceservice.finalizers\"}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:49:57Z INFO InferenceService being deleted {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z DEBUG Deleting InferenceService in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest"} 2026-07-12T09:49:57Z DEBUG Deleting ClusterRoleBinding tensorflow-standard-triton-models-bucket-sa-auth-delegator in namespace tensorflow-standard {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "tensorflow-standard", "name": "inference-prometheus-auth"} 2026-07-12T09:49:57Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "autoscaling": null} 2026-07-12T09:49:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:57Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "3dd27da9-6016-4e4f-b0cf-d2a532e7c889", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "50825312-c793-4947-b491-a8bc85961155", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "50825312-c793-4947-b491-a8bc85961155", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "50825312-c793-4947-b491-a8bc85961155", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "50825312-c793-4947-b491-a8bc85961155", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:57Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "50825312-c793-4947-b491-a8bc85961155", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO Stop InferenceService reconciliation {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "81fe39be-dcea-495b-b3f8-b55fbaf26400", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z DEBUG Triggering RawDeployment cleanup - no active InferenceServices found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "81fe39be-dcea-495b-b3f8-b55fbaf26400", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:57Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "81fe39be-dcea-495b-b3f8-b55fbaf26400", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard", "namespace": "tensorflow-standard"} 2026-07-12T09:49:58Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "81fe39be-dcea-495b-b3f8-b55fbaf26400", "Namespace": "rhoai-model-registries"} 2026-07-12T09:49:58Z DEBUG controllers.ModelRegistryInferenceService Stop ModelRegistry InferenceService reconciliation, ISVC not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"tensorflow-standard-rest","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "tensorflow-standard-rest", "reconcileID": "81fe39be-dcea-495b-b3f8-b55fbaf26400", "InferenceService": "tensorflow-standard-rest", "namespace": "tensorflow-standard"} 2026-07-12T09:49:58Z INFO Reconcile event triggered by serving runtime: triton-rest-runtime 2026-07-12T09:49:58Z INFO No InferenceServices found for Serving Runtime: triton-rest-runtime 2026-07-12T09:49:58Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:49:58Z INFO Multi Node reconciling. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:50:00Z INFO Multi Node reconciled successfully. {"ResourceName": "triton-rest-runtime", "Namespace": "tensorflow-standard"} 2026-07-12T09:50:05Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3f894062-ca43-4b81-87a7-4b727b3842dc", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:50:05Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3f894062-ca43-4b81-87a7-4b727b3842dc", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:50:05Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "3f894062-ca43-4b81-87a7-4b727b3842dc", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:50:05Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a7c0bef1-9537-4f12-b891-f88e45700522", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:50:05Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a7c0bef1-9537-4f12-b891-f88e45700522", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:50:05Z ERROR Reconciler error {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "a7c0bef1-9537-4f12-b891-f88e45700522", "error": "configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: unable to create new content in namespace tensorflow-standard because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:50:05Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "0288dca4-94ed-472b-8fd5-6f526369cffc", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:50:05Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-trusted-ca-bundle", "reconcileID": "0288dca4-94ed-472b-8fd5-6f526369cffc", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:50:05Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "538282fd-bd5d-4e65-a0a0-beb1c2a6d956", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:50:05Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "openshift-service-ca.crt", "reconcileID": "538282fd-bd5d-4e65-a0a0-beb1c2a6d956", "ConfigMap": "openshift-service-ca.crt", "namespace": "tensorflow-standard"} 2026-07-12T09:50:05Z INFO Cert ConfigMap not found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "88755fe2-bf23-449f-96bd-80cbbff8fdfa", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:50:05Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"tensorflow-standard"}, "namespace": "tensorflow-standard", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "88755fe2-bf23-449f-96bd-80cbbff8fdfa", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "tensorflow-standard"} 2026-07-12T09:50:08Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "openshift-service-ca.crt", "reconcileID": "5f351ca1-5112-47cc-b180-ec4852937cdc", "ConfigMap": "openshift-service-ca.crt", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "odh-trusted-ca-bundle", "reconcileID": "0463342d-f937-42f0-864d-56cf2f8b8d7c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "opt-125m-probes", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:50:08Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "odh-trusted-ca-bundle", "reconcileID": "0463342d-f937-42f0-864d-56cf2f8b8d7c", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "openshift-service-ca.crt", "reconcileID": "ae4fd1eb-a867-4fed-a49f-127f2308ea90", "ConfigMap": "openshift-service-ca.crt", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "42e948f8-53f6-4c4b-ab77-bebc61995ba2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "openshift-service-ca.crt", "reconcileID": "87e2f9f3-884a-4695-9601-8cb28548920b", "ConfigMap": "openshift-service-ca.crt", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "odh-trusted-ca-bundle", "reconcileID": "193e2f59-4686-426f-9b87-9070de27ac4e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "opt-125m-probes", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T09:50:08Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "odh-trusted-ca-bundle", "reconcileID": "193e2f59-4686-426f-9b87-9070de27ac4e", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "63c76582-2fbe-43da-8c5b-4c1b68c9925b", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z INFO Reconcile event triggered by serving runtime: vllm-runtime 2026-07-12T09:50:08Z INFO No InferenceServices found for Serving Runtime: vllm-runtime 2026-07-12T09:50:08Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "vllm-runtime", "Namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z INFO Multi Node reconciling. {"ResourceName": "vllm-runtime", "Namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z INFO inferenceservice-resource Defaulting for InferenceService {"name": "opt-125m-probes", "name": "opt-125m-probes"} 2026-07-12T09:50:08Z INFO inferenceservice-resource Validation for InferenceService upon creation {"namespace": "opt-125m-probes", "isvc": "opt-125m-probes"} 2026-07-12T09:50:08Z INFO inferenceservice-resource Checking if namespace is protected {"namespace": "opt-125m-probes", "isvc": "opt-125m-probes", "namespace": "opt-125m-probes", "protectedNamespace": "redhat-ods-applications"} 2026-07-12T09:50:08Z INFO inferenceservice-resource Namespace is not protected {"namespace": "opt-125m-probes", "isvc": "opt-125m-probes"} 2026-07-12T09:50:08Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z ERROR Failed to fetch service for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "InferenceService": "opt-125m-probes"} github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).createDesiredResource /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:110 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawRouteReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_route_reconciler.go:61 github.com/opendatahub-io/odh-model-controller/internal/controller/serving/reconcilers.(*KserveRawInferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/reconcilers/kserve_raw_inferenceservice_reconciler.go:54 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).ReconcileServing /workspace/internal/controller/serving/inferenceservice_controller.go:164 github.com/opendatahub-io/odh-model-controller/internal/controller/serving.(*InferenceServiceReconciler).Reconcile /workspace/internal/controller/serving/inferenceservice_controller.go:173 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:116 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:303 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/controller/controller.go:224 2026-07-12T09:50:08Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG Service not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "create": "opt-125m-probes-metrics"} 2026-07-12T09:50:08Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG ServiceMonitor not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "create": "opt-125m-probes-metrics"} 2026-07-12T09:50:08Z DEBUG ConfigMap not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "create": "opt-125m-probes-metrics-dashboard"} 2026-07-12T09:50:08Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:08Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:08Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:08Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:08Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "Namespace": "rhoai-model-registries"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-probes {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "7705535c-64bf-4c96-97c6-c72df3df045b", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Openshift Route not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "create": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "ConfigMap:": "opt-125m-probes-metrics-dashboard"} 2026-07-12T09:50:09Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:09Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "Namespace": "rhoai-model-registries"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-probes {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "5fd1946a-12f6-4ff5-ae57-f818a4f23e5d", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "update": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "ConfigMap:": "opt-125m-probes-metrics-dashboard"} 2026-07-12T09:50:09Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:09Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "Namespace": "rhoai-model-registries"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-probes {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "baaa56df-179a-4585-83f7-dcae6d09c988", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "update": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "ConfigMap:": "opt-125m-probes-metrics-dashboard"} 2026-07-12T09:50:09Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:09Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "Namespace": "rhoai-model-registries"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-probes {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "6cd8712e-8501-4a2b-af2f-526ab5020599", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO Adding Finalizer {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO Reconciling InferenceService for Kserve in RawDeployment mode {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling ClusterRoleBinding for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG ClusterRoleBinding not found. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Generic Route for Kserve Raw InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed Openshift Route {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "update": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Metrics Service for InferenceService, checking if there are resource for deletion {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed Service {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Reconciling Metrics ServiceMonitor for InferenceService {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetch deployed ServiceMonitor {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG No delta found {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG Successfully fetched ConfigMap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "ConfigMap:": "opt-125m-probes-metrics-dashboard"} 2026-07-12T09:50:09Z DEBUG No delta found in metrics configmap {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler No Prometheus external autoscaling metric found, KEDA resources not required by this InferenceService. Ensuring InferenceService is removed from owner references. {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler Resource not found, skipping owner reference removal {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "kind": "&TypeMeta{Kind:,APIVersion:,}", "namespace": "opt-125m-probes", "name": "inference-prometheus-auth"} 2026-07-12T09:50:09Z DEBUG KserveKEDAReconciler hasPrometheusExternalAutoscalingMetric {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "autoscaling": null} 2026-07-12T09:50:09Z INFO KserveKEDAReconciler Cleaning up KEDA resources in namespace {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService Model Registry Namespace from DataScienceCluster {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "Namespace": "rhoai-model-registries"} 2026-07-12T09:50:09Z INFO controllers.ModelRegistryInferenceService missing model registry specific label, unable to link ISVC to Model Registry, skipping InferenceService: opt-125m-probes {"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"opt-125m-probes","namespace":"opt-125m-probes"}, "namespace": "opt-125m-probes", "name": "opt-125m-probes", "reconcileID": "afce4514-2867-42d0-893a-14e9d36c67ea", "InferenceService": "opt-125m-probes", "namespace": "opt-125m-probes"} 2026-07-12T09:50:13Z INFO Multi Node reconciled successfully. {"ResourceName": "vllm-runtime", "Namespace": "opt-125m-probes"} 2026-07-12T09:50:38Z DEBUG Reconciling EnvoyFilter for Gateway {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "c90523a0-7eb8-4a15-9024-c1e9b76ebf1a", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T09:50:38Z INFO Updating EnvoyFilter {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "c90523a0-7eb8-4a15-9024-c1e9b76ebf1a", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress", "name": "maas-default-gateway-authn-ssl"} 2026-07-12T09:50:38Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"maas-default-gateway","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "maas-default-gateway", "reconcileID": "c90523a0-7eb8-4a15-9024-c1e9b76ebf1a", "Gateway": "maas-default-gateway", "namespace": "openshift-ingress"} 2026-07-12T09:51:37Z INFO No monitoring namespace detected, skipping monitoring reconciliation. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:51:37Z INFO Multi Node reconciling. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:51:39Z INFO ray ca tls secret modified so ray tls cert need to be updated to sync ca.crt {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T09:51:39Z INFO Multi Node reconciled successfully. {"ResourceName": "ray-ca-tls", "Namespace": "redhat-ods-applications"} 2026-07-12T10:03:40Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ba61e35b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ba61e35b", "reconcileID": "4b35a61b-6712-4c44-8ae8-c62859886845", "Gateway": "e2e-aigw-ba61e35b", "namespace": "openshift-ingress"} 2026-07-12T10:03:40Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ba61e35b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ba61e35b", "reconcileID": "dd9bde5c-e116-48c0-933d-dc6074693bc2", "Gateway": "e2e-aigw-ba61e35b", "namespace": "openshift-ingress"} 2026-07-12T10:03:40Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ba61e35b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ba61e35b", "reconcileID": "e4066afd-84bf-4768-aeff-b66947184bc9", "Gateway": "e2e-aigw-ba61e35b", "namespace": "openshift-ingress"} 2026-07-12T10:03:40Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ba61e35b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ba61e35b", "reconcileID": "493dc9d9-c9fc-442f-83b9-9bfc251b5252", "Gateway": "e2e-aigw-ba61e35b", "namespace": "openshift-ingress"} 2026-07-12T10:03:40Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "odh-trusted-ca-bundle", "reconcileID": "c430c3c7-765d-4d9c-857a-f7a4f0475cb0", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ba61e35b", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T10:03:40Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "odh-trusted-ca-bundle", "reconcileID": "c430c3c7-765d-4d9c-857a-f7a4f0475cb0", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ba61e35b"} 2026-07-12T10:03:40Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "48b5eaed-c2b9-40c3-91b9-e5aa7c59a858", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ba61e35b"} 2026-07-12T10:03:41Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "openshift-service-ca.crt", "reconcileID": "039a976e-feec-46d1-a025-9f0299bb3cf6", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-ba61e35b"} 2026-07-12T10:03:41Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "openshift-service-ca.crt", "reconcileID": "3102f03e-789c-4be3-9edb-1c064c745dbe", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-ba61e35b", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T10:03:41Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "openshift-service-ca.crt", "reconcileID": "3102f03e-789c-4be3-9edb-1c064c745dbe", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-ba61e35b"} 2026-07-12T10:03:41Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "ecbc17ac-22da-44f9-a003-e2903ab1b864", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ba61e35b"} 2026-07-12T10:03:41Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "odh-trusted-ca-bundle", "reconcileID": "bfbf5688-e513-479a-8285-426a17f5b232", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ba61e35b", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T10:03:41Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "odh-trusted-ca-bundle", "reconcileID": "bfbf5688-e513-479a-8285-426a17f5b232", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ba61e35b"} 2026-07-12T10:03:41Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "openshift-service-ca.crt", "reconcileID": "4dce0966-eabe-46f2-b9cd-58e059b99d35", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-ba61e35b"} 2026-07-12T10:03:41Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-ba61e35b"}, "namespace": "ai-tenant-e2e-aigw-ba61e35b", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "afaea0b4-5c61-4b4e-badf-9447e66d0d47", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ba61e35b"} 2026-07-12T10:03:41Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ba61e35b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ba61e35b", "reconcileID": "46d0efdd-0ce9-4fde-8969-3626e40b861a", "Gateway": "e2e-aigw-ba61e35b", "namespace": "openshift-ingress"} 2026-07-12T10:03:42Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "openshift-service-ca.crt", "reconcileID": "14947c2a-4a0a-4306-8576-dd0281946b98", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-applications"} 2026-07-12T10:03:42Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ba61e35b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ba61e35b", "reconcileID": "12187e39-48ff-496c-a872-7acce001068f", "Gateway": "e2e-aigw-ba61e35b", "namespace": "openshift-ingress"} 2026-07-12T10:03:42Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ba61e35b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ba61e35b", "reconcileID": "08e80b28-0d2d-4dbc-abcb-ee0284a3b8a1", "Gateway": "e2e-aigw-ba61e35b", "namespace": "openshift-ingress"} 2026-07-12T10:03:44Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ba61e35b","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ba61e35b", "reconcileID": "4e517147-49d8-4cf0-ab69-d83e030224da", "Gateway": "e2e-aigw-ba61e35b", "namespace": "openshift-ingress"} 2026-07-12T10:03:46Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ef12e752","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ef12e752", "reconcileID": "b7b1e401-44e8-4602-a0e7-46aa276880a6", "Gateway": "e2e-aigw-ef12e752", "namespace": "openshift-ingress"} 2026-07-12T10:03:46Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ef12e752","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ef12e752", "reconcileID": "380c37ad-1d84-475b-86fa-92dee548cacd", "Gateway": "e2e-aigw-ef12e752", "namespace": "openshift-ingress"} 2026-07-12T10:03:46Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ef12e752","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ef12e752", "reconcileID": "ca46ed7e-ff8e-40a0-9073-5ad3b02eb807", "Gateway": "e2e-aigw-ef12e752", "namespace": "openshift-ingress"} 2026-07-12T10:03:46Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ef12e752","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ef12e752", "reconcileID": "36dc8a00-8467-48b1-aed1-5695de416ccd", "Gateway": "e2e-aigw-ef12e752", "namespace": "openshift-ingress"} 2026-07-12T10:03:46Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-ef12e752"}, "namespace": "ai-tenant-e2e-aigw-ef12e752", "name": "openshift-service-ca.crt", "reconcileID": "db8f65f6-f559-4e15-89bb-43c9b3f33c99", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-ef12e752"} 2026-07-12T10:03:46Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-ef12e752"}, "namespace": "ai-tenant-e2e-aigw-ef12e752", "name": "odh-trusted-ca-bundle", "reconcileID": "33110d6b-1797-462c-a5fc-e9e121c1875b", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ef12e752", "create": "odh-kserve-custom-ca-bundle"} 2026-07-12T10:03:46Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-ef12e752"}, "namespace": "ai-tenant-e2e-aigw-ef12e752", "name": "odh-trusted-ca-bundle", "reconcileID": "33110d6b-1797-462c-a5fc-e9e121c1875b", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ef12e752"} 2026-07-12T10:03:46Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"ai-tenant-e2e-aigw-ef12e752"}, "namespace": "ai-tenant-e2e-aigw-ef12e752", "name": "openshift-service-ca.crt", "reconcileID": "7c154e51-1bdb-4814-a33f-090f1c39c77e", "ConfigMap": "openshift-service-ca.crt", "namespace": "ai-tenant-e2e-aigw-ef12e752"} 2026-07-12T10:03:46Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-ef12e752"}, "namespace": "ai-tenant-e2e-aigw-ef12e752", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "92d82c2c-159e-497a-86f2-4b17c04d3ed2", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ef12e752"} 2026-07-12T10:03:46Z DEBUG Delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-ef12e752"}, "namespace": "ai-tenant-e2e-aigw-ef12e752", "name": "odh-trusted-ca-bundle", "reconcileID": "6e223e54-cec1-45a4-a0d7-429050f33e4d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ef12e752", "update": "odh-kserve-custom-ca-bundle"} 2026-07-12T10:03:46Z DEBUG Deleted the storage-config Secret to update the custom cert {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-trusted-ca-bundle","namespace":"ai-tenant-e2e-aigw-ef12e752"}, "namespace": "ai-tenant-e2e-aigw-ef12e752", "name": "odh-trusted-ca-bundle", "reconcileID": "6e223e54-cec1-45a4-a0d7-429050f33e4d", "ConfigMap": "odh-trusted-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ef12e752"} 2026-07-12T10:03:46Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"odh-kserve-custom-ca-bundle","namespace":"ai-tenant-e2e-aigw-ef12e752"}, "namespace": "ai-tenant-e2e-aigw-ef12e752", "name": "odh-kserve-custom-ca-bundle", "reconcileID": "e3d1832a-5351-492d-bd2b-b7a210f1e7cf", "ConfigMap": "odh-kserve-custom-ca-bundle", "namespace": "ai-tenant-e2e-aigw-ef12e752"} 2026-07-12T10:03:46Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ef12e752","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ef12e752", "reconcileID": "696b07df-e374-429a-bd2b-786fe24b4d41", "Gateway": "e2e-aigw-ef12e752", "namespace": "openshift-ingress"} 2026-07-12T10:03:47Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "openshift-service-ca.crt", "reconcileID": "78933d5f-eeff-4d19-9f36-6d1993cfd449", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-applications"} 2026-07-12T10:03:47Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ef12e752","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ef12e752", "reconcileID": "900372ab-e77f-4807-882d-53fc5803e8e2", "Gateway": "e2e-aigw-ef12e752", "namespace": "openshift-ingress"} 2026-07-12T10:03:48Z DEBUG Successfully reconciled Gateway resources {"controller": "gateway-auth-bootstrap", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"e2e-aigw-ef12e752","namespace":"openshift-ingress"}, "namespace": "openshift-ingress", "name": "e2e-aigw-ef12e752", "reconcileID": "af39a8ee-3e78-430d-8182-a2c0fc6b6634", "Gateway": "e2e-aigw-ef12e752", "namespace": "openshift-ingress"} 2026-07-12T10:03:55Z DEBUG No delta found {"controller": "core-configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"openshift-service-ca.crt","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "openshift-service-ca.crt", "reconcileID": "bddd5aaa-c051-41f0-be63-2c13c099e6d7", "ConfigMap": "openshift-service-ca.crt", "namespace": "redhat-ods-applications"} ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-7c7d9467f5-pmns6.log ==================== 2026/07/12 09:37:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41674 - 200 55B in 51.68µs 2026/07/12 09:37:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41688 - 200 55B in 41.985µs 2026/07/12 09:37:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42270 - 200 55B in 39.04µs 2026/07/12 09:38:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42280 - 200 55B in 38.868µs 2026/07/12 09:38:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60774 - 200 55B in 39.176µs 2026/07/12 09:38:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60790 - 200 55B in 38.839µs 2026/07/12 09:38:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41676 - 200 55B in 48.909µs 2026/07/12 09:38:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41694 - 200 55B in 47.991µs 2026/07/12 09:38:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40860 - 200 55B in 39.777µs 2026/07/12 09:38:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40890 - 200 55B in 46.816µs 2026/07/12 09:38:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58806 - 200 55B in 37.127µs 2026/07/12 09:38:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58830 - 200 55B in 38.875µs 2026/07/12 09:38:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40518 - 200 55B in 43.969µs 2026/07/12 09:38:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40540 - 200 55B in 40.237µs 2026/07/12 09:38:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35408 - 200 55B in 38.26µs 2026/07/12 09:39:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35412 - 200 55B in 37.289µs 2026/07/12 09:39:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56562 - 200 55B in 43.847µs 2026/07/12 09:39:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56594 - 200 55B in 41.273µs 2026/07/12 09:39:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54024 - 200 55B in 23.748µs 2026/07/12 09:39:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54052 - 200 55B in 24.008µs 2026/07/12 09:39:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52832 - 200 55B in 41.067µs 2026/07/12 09:39:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52850 - 200 55B in 22.575µs 2026/07/12 09:39:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46994 - 200 55B in 39.767µs 2026/07/12 09:39:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47002 - 200 55B in 40.512µs 2026/07/12 09:39:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51308 - 200 55B in 91.246µs 2026/07/12 09:39:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51322 - 200 55B in 37.965µs 2026/07/12 09:39:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54514 - 200 55B in 43.788µs 2026/07/12 09:40:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54532 - 200 55B in 46.057µs 2026/07/12 09:40:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46726 - 200 55B in 40.225µs 2026/07/12 09:40:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46740 - 200 55B in 39.345µs 2026/07/12 09:40:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44886 - 200 55B in 45.024µs 2026/07/12 09:40:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44908 - 200 55B in 51.173µs 2026/07/12 09:40:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54730 - 200 55B in 38.282µs 2026/07/12 09:40:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54752 - 200 55B in 38.075µs 2026/07/12 09:40:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40826 - 200 55B in 57.328µs 2026/07/12 09:40:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40836 - 200 55B in 39.505µs 2026/07/12 09:40:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33774 - 200 55B in 37.472µs 2026/07/12 09:40:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33792 - 200 55B in 39.944µs 2026/07/12 09:40:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46518 - 200 55B in 37.883µs 2026/07/12 09:41:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46540 - 200 55B in 39.498µs 2026/07/12 09:41:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42428 - 200 55B in 37.869µs 2026/07/12 09:41:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42438 - 200 55B in 37.707µs 2026/07/12 09:41:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33596 - 200 55B in 40.325µs 2026/07/12 09:41:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33616 - 200 55B in 23.771µs 2026/07/12 09:41:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50176 - 200 55B in 52.137µs 2026/07/12 09:41:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50194 - 200 55B in 39.91µs 2026/07/12 09:41:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40040 - 200 55B in 39.415µs 2026/07/12 09:41:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40052 - 200 55B in 39.054µs 2026/07/12 09:41:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57138 - 200 55B in 21.719µs 2026/07/12 09:41:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:57166 - 200 55B in 38.258µs 2026/07/12 09:41:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60018 - 200 55B in 40.444µs 2026/07/12 09:42:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60032 - 200 55B in 38.001µs 2026/07/12 09:42:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40646 - 200 55B in 48.297µs 2026/07/12 09:42:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40660 - 200 55B in 47.276µs 2026/07/12 09:42:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37644 - 200 55B in 44.283µs 2026/07/12 09:42:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37660 - 200 55B in 41.124µs 2026/07/12 09:42:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50672 - 200 55B in 48.64µs 2026/07/12 09:42:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50692 - 200 55B in 56.186µs 2026/07/12 09:42:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40500 - 200 55B in 45.909µs 2026/07/12 09:42:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40518 - 200 55B in 39.786µs 2026/07/12 09:42:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55636 - 200 55B in 41.244µs 2026/07/12 09:42:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55642 - 200 55B in 38.309µs 2026/07/12 09:42:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36472 - 200 55B in 38.049µs 2026/07/12 09:43:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36488 - 200 55B in 38.327µs 2026/07/12 09:43:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41988 - 200 55B in 37.552µs 2026/07/12 09:43:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42004 - 200 55B in 38.373µs 2026/07/12 09:43:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54726 - 200 55B in 39.848µs 2026/07/12 09:43:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54738 - 200 55B in 42.662µs 2026/07/12 09:43:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58780 - 200 55B in 37.977µs 2026/07/12 09:43:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58792 - 200 55B in 29.667µs 2026/07/12 09:43:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36044 - 200 55B in 39.653µs 2026/07/12 09:43:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36074 - 200 55B in 39.447µs 2026/07/12 09:43:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54724 - 200 55B in 38.451µs 2026/07/12 09:43:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54734 - 200 55B in 37.82µs 2026/07/12 09:43:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46158 - 200 55B in 37.629µs 2026/07/12 09:44:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46174 - 200 55B in 42.343µs 2026/07/12 09:44:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46386 - 200 55B in 57.866µs 2026/07/12 09:44:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46394 - 200 55B in 39.834µs 2026/07/12 09:44:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47992 - 200 55B in 40.363µs 2026/07/12 09:44:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48018 - 200 55B in 37.075µs 2026/07/12 09:44:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43104 - 200 55B in 46.927µs 2026/07/12 09:44:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43110 - 200 55B in 43.841µs 2026/07/12 09:44:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43344 - 200 55B in 40.324µs 2026/07/12 09:44:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:43366 - 200 55B in 39.76µs 2026/07/12 09:44:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34114 - 200 55B in 43.258µs 2026/07/12 09:44:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34132 - 200 55B in 40.886µs 2026/07/12 09:44:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33400 - 200 55B in 37.506µs 2026/07/12 09:45:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33422 - 200 55B in 37.769µs 2026/07/12 09:45:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38798 - 200 55B in 73.78µs 2026/07/12 09:45:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38822 - 200 55B in 43.523µs 2026/07/12 09:45:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40888 - 200 55B in 24.671µs 2026/07/12 09:45:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40898 - 200 55B in 23.008µs 2026/07/12 09:45:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41946 - 200 55B in 39.862µs 2026/07/12 09:45:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41954 - 200 55B in 44.384µs 2026/07/12 09:45:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40110 - 200 55B in 38.642µs 2026/07/12 09:45:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40124 - 200 55B in 36.78µs 2026/07/12 09:45:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36532 - 200 55B in 41.015µs 2026/07/12 09:45:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36540 - 200 55B in 38.53µs 2026/07/12 09:45:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55104 - 200 55B in 63.354µs 2026/07/12 09:46:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55118 - 200 55B in 39.15µs 2026/07/12 09:46:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59256 - 200 55B in 37.553µs 2026/07/12 09:46:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59266 - 200 55B in 38.594µs 2026/07/12 09:46:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44804 - 200 55B in 45.143µs 2026/07/12 09:46:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44822 - 200 55B in 37.291µs 2026/07/12 09:46:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45788 - 200 55B in 55.332µs 2026/07/12 09:46:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45810 - 200 55B in 59.528µs 2026/07/12 09:46:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48326 - 200 55B in 39.729µs 2026/07/12 09:46:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48336 - 200 55B in 39.085µs 2026/07/12 09:46:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44296 - 200 55B in 46.853µs 2026/07/12 09:46:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44314 - 200 55B in 42.534µs 2026/07/12 09:46:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36120 - 200 55B in 45.4µs 2026/07/12 09:47:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36140 - 200 55B in 39.379µs 2026/07/12 09:47:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53364 - 200 55B in 43.882µs 2026/07/12 09:47:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53380 - 200 55B in 39.303µs 2026/07/12 09:47:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39974 - 200 55B in 39.786µs 2026/07/12 09:47:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39998 - 200 55B in 41.531µs 2026/07/12 09:47:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36174 - 200 55B in 26.137µs 2026/07/12 09:47:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36196 - 200 55B in 37.706µs 2026/07/12 09:47:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45356 - 200 55B in 39.191µs 2026/07/12 09:47:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45378 - 200 55B in 60.487µs 2026/07/12 09:47:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49848 - 200 55B in 37.201µs 2026/07/12 09:47:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49864 - 200 55B in 40.962µs 2026/07/12 09:47:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49246 - 200 55B in 44.148µs 2026/07/12 09:48:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49272 - 200 55B in 46.852µs 2026/07/12 09:48:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34558 - 200 55B in 38.627µs 2026/07/12 09:48:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34582 - 200 55B in 42.909µs 2026/07/12 09:48:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49552 - 200 55B in 41.744µs 2026/07/12 09:48:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49568 - 200 55B in 38.55µs 2026/07/12 09:48:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60066 - 200 55B in 36.515µs 2026/07/12 09:48:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60086 - 200 55B in 47.712µs 2026/07/12 09:48:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54662 - 200 55B in 58.617µs 2026/07/12 09:48:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54674 - 200 55B in 40.681µs 2026/07/12 09:48:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39678 - 200 55B in 39.178µs 2026/07/12 09:48:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39688 - 200 55B in 39.23µs 2026/07/12 09:48:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42286 - 200 55B in 38.741µs 2026/07/12 09:49:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42306 - 200 55B in 37.169µs 2026/07/12 09:49:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41638 - 200 55B in 46.295µs 2026/07/12 09:49:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41648 - 200 55B in 36.476µs 2026/07/12 09:49:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54618 - 200 55B in 51.744µs 2026/07/12 09:49:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54640 - 200 55B in 23.042µs 2026/07/12 09:49:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60756 - 200 55B in 23.27µs 2026/07/12 09:49:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60776 - 200 55B in 63.981µs 2026/07/12 09:49:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41544 - 200 55B in 37.036µs 2026/07/12 09:49:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41554 - 200 55B in 38.028µs 2026/07/12 09:49:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45080 - 200 55B in 45.795µs 2026/07/12 09:49:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45104 - 200 55B in 37.782µs 2026/07/12 09:49:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34306 - 200 55B in 39.642µs 2026/07/12 09:50:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34320 - 200 55B in 38.81µs 2026/07/12 09:50:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42298 - 200 55B in 61.286µs 2026/07/12 09:50:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42312 - 200 55B in 41.731µs 2026/07/12 09:50:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41938 - 200 55B in 40.091µs 2026/07/12 09:50:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41946 - 200 55B in 37.474µs 2026/07/12 09:50:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45826 - 200 55B in 38.324µs 2026/07/12 09:50:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45856 - 200 55B in 45.099µs 2026/07/12 09:50:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49518 - 200 55B in 37.517µs 2026/07/12 09:50:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49538 - 200 55B in 69.069µs 2026/07/12 09:50:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58054 - 200 55B in 39.153µs 2026/07/12 09:50:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58072 - 200 55B in 38.066µs 2026/07/12 09:50:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39830 - 200 55B in 39.317µs 2026/07/12 09:51:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39842 - 200 55B in 38.937µs 2026/07/12 09:51:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41422 - 200 55B in 40.285µs 2026/07/12 09:51:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41438 - 200 55B in 36.612µs 2026/07/12 09:51:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60140 - 200 55B in 39.511µs 2026/07/12 09:51:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60156 - 200 55B in 24.104µs 2026/07/12 09:51:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52090 - 200 55B in 41.066µs 2026/07/12 09:51:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52106 - 200 55B in 40.89µs 2026/07/12 09:51:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36698 - 200 55B in 38.797µs 2026/07/12 09:51:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36706 - 200 55B in 40.592µs 2026/07/12 09:51:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48176 - 200 55B in 22.197µs 2026/07/12 09:51:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48200 - 200 55B in 39.318µs 2026/07/12 09:51:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47066 - 200 55B in 37.53µs 2026/07/12 09:52:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47086 - 200 55B in 40.003µs 2026/07/12 09:52:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33728 - 200 55B in 39.78µs 2026/07/12 09:52:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33756 - 200 55B in 37.095µs 2026/07/12 09:52:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48816 - 200 55B in 59.676µs 2026/07/12 09:52:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48838 - 200 55B in 37.638µs 2026/07/12 09:52:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45020 - 200 55B in 37.549µs 2026/07/12 09:52:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45044 - 200 55B in 42.12µs 2026/07/12 09:52:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50232 - 200 55B in 63.37µs 2026/07/12 09:52:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50248 - 200 55B in 37.139µs 2026/07/12 09:52:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36604 - 200 55B in 42.271µs 2026/07/12 09:52:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36608 - 200 55B in 57.627µs 2026/07/12 09:52:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37100 - 200 55B in 38.557µs 2026/07/12 09:53:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37110 - 200 55B in 64.259µs 2026/07/12 09:53:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45746 - 200 55B in 37.477µs 2026/07/12 09:53:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45768 - 200 55B in 40.476µs 2026/07/12 09:53:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56616 - 200 55B in 31.168µs 2026/07/12 09:53:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56634 - 200 55B in 22.299µs 2026/07/12 09:53:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47304 - 200 55B in 39.595µs 2026/07/12 09:53:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47322 - 200 55B in 40.198µs 2026/07/12 09:53:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60520 - 200 55B in 39.475µs 2026/07/12 09:53:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60524 - 200 55B in 21.442µs 2026/07/12 09:53:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52430 - 200 55B in 37.389µs 2026/07/12 09:53:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52444 - 200 55B in 38.542µs 2026/07/12 09:53:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51278 - 200 55B in 39.246µs 2026/07/12 09:54:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51292 - 200 55B in 39.652µs 2026/07/12 09:54:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41854 - 200 55B in 38.174µs 2026/07/12 09:54:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41864 - 200 55B in 37.335µs 2026/07/12 09:54:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48966 - 200 55B in 39.519µs 2026/07/12 09:54:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48984 - 200 55B in 42.002µs 2026/07/12 09:54:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51198 - 200 55B in 39.926µs 2026/07/12 09:54:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51204 - 200 55B in 58.054µs 2026/07/12 09:54:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58542 - 200 55B in 38.874µs 2026/07/12 09:54:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58560 - 200 55B in 39.956µs 2026/07/12 09:54:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53158 - 200 55B in 38.464µs 2026/07/12 09:54:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53170 - 200 55B in 50.93µs 2026/07/12 09:54:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33112 - 200 55B in 37.224µs 2026/07/12 09:55:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:33128 - 200 55B in 36.673µs 2026/07/12 09:55:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40720 - 200 55B in 43.368µs 2026/07/12 09:55:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:40746 - 200 55B in 37.086µs 2026/07/12 09:55:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38148 - 200 55B in 22.938µs 2026/07/12 09:55:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38154 - 200 55B in 37.163µs 2026/07/12 09:55:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34074 - 200 55B in 21.835µs 2026/07/12 09:55:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:34100 - 200 55B in 38.392µs 2026/07/12 09:55:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41402 - 200 55B in 39.376µs 2026/07/12 09:55:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41418 - 200 55B in 22.602µs 2026/07/12 09:55:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50334 - 200 55B in 50.595µs 2026/07/12 09:55:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50352 - 200 55B in 39.773µs 2026/07/12 09:55:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51882 - 200 55B in 40.699µs 2026/07/12 09:56:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51894 - 200 55B in 38.38µs 2026/07/12 09:56:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52146 - 200 55B in 38.091µs 2026/07/12 09:56:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52176 - 200 55B in 61.303µs 2026/07/12 09:56:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50602 - 200 55B in 41.813µs 2026/07/12 09:56:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:50628 - 200 55B in 37.097µs 2026/07/12 09:56:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37010 - 200 55B in 38.341µs 2026/07/12 09:56:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37042 - 200 55B in 42.287µs 2026/07/12 09:56:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54620 - 200 55B in 38.524µs 2026/07/12 09:56:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:54650 - 200 55B in 36.389µs 2026/07/12 09:56:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60856 - 200 55B in 42.523µs 2026/07/12 09:56:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60872 - 200 55B in 39.936µs 2026/07/12 09:56:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39690 - 200 55B in 54.619µs 2026/07/12 09:57:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39710 - 200 55B in 37.704µs 2026/07/12 09:57:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59970 - 200 55B in 47.4µs 2026/07/12 09:57:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59992 - 200 55B in 37.017µs 2026/07/12 09:57:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52830 - 200 55B in 22.773µs 2026/07/12 09:57:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52856 - 200 55B in 23.326µs 2026/07/12 09:57:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44882 - 200 55B in 38.05µs 2026/07/12 09:57:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44896 - 200 55B in 38.802µs 2026/07/12 09:57:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37414 - 200 55B in 42.041µs 2026/07/12 09:57:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37434 - 200 55B in 46.537µs 2026/07/12 09:57:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39608 - 200 55B in 23.318µs 2026/07/12 09:57:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39640 - 200 55B in 39.263µs 2026/07/12 09:57:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44200 - 200 55B in 35.876µs 2026/07/12 09:58:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44212 - 200 55B in 38.524µs 2026/07/12 09:58:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44492 - 200 55B in 50.063µs 2026/07/12 09:58:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44504 - 200 55B in 47.767µs 2026/07/12 09:58:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:32982 - 200 55B in 45.858µs 2026/07/12 09:58:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:32998 - 200 55B in 37.345µs 2026/07/12 09:58:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55998 - 200 55B in 36.095µs 2026/07/12 09:58:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56014 - 200 55B in 37.21µs 2026/07/12 09:58:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53692 - 200 55B in 47.737µs 2026/07/12 09:58:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:53716 - 200 55B in 37.7µs 2026/07/12 09:58:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48462 - 200 55B in 67.434µs 2026/07/12 09:58:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48474 - 200 55B in 42.765µs 2026/07/12 09:58:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48236 - 200 55B in 37.275µs 2026/07/12 09:59:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48254 - 200 55B in 40.05µs 2026/07/12 09:59:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38528 - 200 55B in 36.948µs 2026/07/12 09:59:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38548 - 200 55B in 39.121µs 2026/07/12 09:59:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:32866 - 200 55B in 50.391µs 2026/07/12 09:59:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:32878 - 200 55B in 23.46µs 2026/07/12 09:59:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51100 - 200 55B in 22.055µs 2026/07/12 09:59:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51112 - 200 55B in 38.149µs 2026/07/12 09:59:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58322 - 200 55B in 39.682µs 2026/07/12 09:59:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58338 - 200 55B in 64.778µs 2026/07/12 09:59:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44994 - 200 55B in 36.981µs 2026/07/12 09:59:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:45018 - 200 55B in 38.734µs 2026/07/12 09:59:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44562 - 200 55B in 46.394µs 2026/07/12 10:00:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44582 - 200 55B in 38.386µs 2026/07/12 10:00:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46450 - 200 55B in 38.461µs 2026/07/12 10:00:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:46458 - 200 55B in 43.959µs 2026/07/12 10:00:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58682 - 200 55B in 41.191µs 2026/07/12 10:00:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:58700 - 200 55B in 36.632µs 2026/07/12 10:00:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49480 - 200 55B in 36.889µs 2026/07/12 10:00:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49506 - 200 55B in 41.915µs 2026/07/12 10:00:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49450 - 200 55B in 48.078µs 2026/07/12 10:00:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:49470 - 200 55B in 47.997µs 2026/07/12 10:00:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39874 - 200 55B in 49.868µs 2026/07/12 10:00:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:39896 - 200 55B in 59.191µs 2026/07/12 10:00:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42118 - 200 55B in 36.963µs 2026/07/12 10:01:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42138 - 200 55B in 49.527µs 2026/07/12 10:01:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44066 - 200 55B in 40.998µs 2026/07/12 10:01:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:44088 - 200 55B in 38.113µs 2026/07/12 10:01:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36818 - 200 55B in 43.703µs 2026/07/12 10:01:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:36838 - 200 55B in 23.077µs 2026/07/12 10:01:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35072 - 200 55B in 22.308µs 2026/07/12 10:01:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35094 - 200 55B in 38.038µs 2026/07/12 10:01:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38322 - 200 55B in 75.988µs 2026/07/12 10:01:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38340 - 200 55B in 38.541µs 2026/07/12 10:01:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42184 - 200 55B in 52.37µs 2026/07/12 10:01:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42194 - 200 55B in 38.416µs 2026/07/12 10:01:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35162 - 200 55B in 36.731µs 2026/07/12 10:02:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35188 - 200 55B in 50.666µs 2026/07/12 10:02:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48490 - 200 55B in 37.865µs 2026/07/12 10:02:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:48520 - 200 55B in 38.565µs 2026/07/12 10:02:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38606 - 200 55B in 61.368µs 2026/07/12 10:02:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:38624 - 200 55B in 37.92µs 2026/07/12 10:02:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60052 - 200 55B in 37.579µs 2026/07/12 10:02:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:60070 - 200 55B in 38.499µs 2026/07/12 10:02:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37108 - 200 55B in 47.532µs 2026/07/12 10:02:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37138 - 200 55B in 39.235µs 2026/07/12 10:02:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35412 - 200 55B in 43.327µs 2026/07/12 10:02:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:35422 - 200 55B in 38.622µs 2026/07/12 10:02:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:51990 - 200 55B in 36.607µs 2026/07/12 10:03:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52014 - 200 55B in 42.993µs 2026/07/12 10:03:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41214 - 200 55B in 38.117µs 2026/07/12 10:03:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:41240 - 200 55B in 38.584µs 2026/07/12 10:03:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52750 - 200 55B in 43.533µs 2026/07/12 10:03:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:52772 - 200 55B in 22.48µs 2026/07/12 10:03:25 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56376 - 200 55B in 22.045µs 2026/07/12 10:03:30 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:56400 - 200 55B in 39.789µs 2026/07/12 10:03:35 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59030 - 200 55B in 36.586µs 2026/07/12 10:03:40 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:59042 - 200 55B in 76.738µs 2026/07/12 10:03:45 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37094 - 200 55B in 37.374µs 2026/07/12 10:03:50 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:37104 - 200 55B in 39.985µs 2026/07/12 10:03:55 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47846 - 200 55B in 37.728µs 2026/07/12 10:04:00 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:47866 - 200 55B in 67.414µs 2026/07/12 10:04:05 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42414 - 200 55B in 36.788µs 2026/07/12 10:04:10 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:42422 - 200 55B in 35.956µs 2026/07/12 10:04:15 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55604 - 200 55B in 40.345µs 2026/07/12 10:04:20 "GET http://10.130.0.27:8080/readyz HTTP/1.1" from 10.130.0.2:55614 - 200 55B in 39.214µs ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-566f8bdddc-66b9q.log ====================